Hi Igniters,
I've detected some new issue on TeamCity to be handled. You are more than
welcomed to help.
*Test with high flaky rate in master
CacheExchangeMergeTest.testMergeStartRandomClientsServers
Val, my understanding about it was exactly the same as yours, but, again, I
heard a different opinion.
But nevertheless, binary protocol should not be about objects, records aka
tuples are the best varii, simple and powerful.
As for me, I want to take part in implementing python and golang thin
Ivan,
KV view does work over the tuples. Nested objects and arbitrary structures
can be stored as blobs. So if you need a basic KV cache, you can always
create a table with two blob fields - one for key and one for value - and
store anything there.
-Val
On Thu, Jul 1, 2021 at 9:55 AM Ivan
An extra argument for IgniteSystemProperty sounds reasonable.
-Val
On Thu, Jul 1, 2021 at 10:04 AM Ivan Daschinsky wrote:
> Ok, this can be excluded using blocklist-jvm-params.properties or just by
> providing and extra arg to annotation, as I have just suggested
>
> чт, 1 июл. 2021 г., 19:51
Ok, this can be excluded using blocklist-jvm-params.properties or just by
providing and extra arg to annotation, as I have just suggested
чт, 1 июл. 2021 г., 19:51 Valentin Kulichenko :
> Ivan,
>
> IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> (e.g. IGNITE_CONFIG_URL) are
Ilya,
I don't think this is the best approach because there are so many
properties that contain so many different types of information. "All or
nothing" doesn't really fit here. We want to have the ability to exclude
sensitive information, but still print out internal settings that are
useful for
Why not mask the default known sensitive options using a blocklist?
On Thu, 1 Jul 2021, 22:24 Shishkov Ilya, wrote:
> Folks,
>
> > Maybe we should add an extra JVM option (e.g.
> IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default,
> > but if set to 'true' then #ackVmOptions will print
Val, am I right, that kv view over the tuples is just simple mapping from
POJO to tuple? No collections, no nested objects? I have discussed this in
private with Igor and Pavel and they told me different info.
чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko :
> Ivan,
>
> I was answering your
Folks,
> Maybe we should add an extra JVM option (e.g.
IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default,
> but if set to 'true' then #ackVmOptions will print VM arguments even if
sensitive data is restricted?
What do you think about an extra JVM option?
чт, 1 июл. 2021 г. в 19:51,
Ivan,
IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
(e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to be
sensitive.
Once again - I would exclude any property that can contain
We can add add an extra param in annotation, that blocks param to be
printed, just set it to false by default and block it wheb set to true
чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> What if we allowed a blocklist of parameters that are never printed?
>
> On Thu, 1 Jul 2021, 22:06 Valentin
What if we allowed a blocklist of parameters that are never printed?
On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
valentin.kuliche...@gmail.com> wrote:
> Not all of them are OK to be printed out. At the very least, we should have
> a mechanism to exclude some of them. I would still go with
Hi,
Maybe we should add an extra JVM option (e.g.
IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default, but if set to
'true' then #ackVmOptions will print VM arguments even if sensitive data is
restricted?
чт, 1 июл. 2021 г. в 19:41, Ivan Daschinsky :
> > Not all of them are OK to be
Ivan,
I was answering your question about the KV API. The API I provided has been
discussed and agreed upon. One of the goals of the protocol is to implement
this API, so it should give you a clear idea of what we're looking for.
Of course, I agree with you that the protocol should be simple and
> Not all of them are OK to be printed out
Could you give an example please? As for me, all of them are pretty harmless
чт, 1 июл. 2021 г., 19:36 Valentin Kulichenko :
> Not all of them are OK to be printed out. At the very least, we should have
> a mechanism to exclude some of them. I would
Andrey, yep, you are right. This was just a quick idea. As for me, I just
don't want to repeat the same problem with compactFooter in thin client api
of ignite 2.x.
чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov :
> >
> > I suppose that we should describe this more verbose and explicit. I
> >
Not all of them are OK to be printed out. At the very least, we should have
a mechanism to exclude some of them. I would still go with opt-in rather
than opt-out though, but I guess that is up to a discussion.
-Val
On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky wrote:
> This is security
This is security through obscurity, an obvious and a well-known anti
pattern. I suppose that printing jvm options, that is registered by
@IgniteSystemProperty annotation is an ideal variant
чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko :
> Folks,
>
> *Anything* that a user provides to the system
Val, when we think about binary protocol, using java interface as an
example is a not good idea. Please, see a tarantool binary api as an
example. Implementor of the client can write it immediately (1)
[1] --
https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/
чт, 1 июл.
Folks,
*Anything* that a user provides to the system can potentially be considered
sensitive information. This includes the VM arguments. We can't predict
what exactly one can put there, so let's not make assumptions.
When dealing with security, we should be as conservative as possible. That
>
> I suppose that we should describe this more verbose and explicit. I
> nevertheless suggest to also consider writing values this way:
> - arr of fields names (if name is missed, corresponding field is nil)
> - arr of rows (row as array, length equal to fields array)
Ivan,
I think GET and PUT
Igor, as you can see from the code, we read data from msgpack and build the
tuple from scratch. Then we serialize this tuple again when send it to
backups. I presume this is absolutely unnecessary
чт, 1 июл. 2021 г., 18:42 Igor Sapego :
> Ivan, what are extra serde steps you are talking about?
>
Ivan,
Regarding the API, please take a look at this package:
https://github.com/apache/ignite-3/tree/main/modules/api/src/main/java/org/apache/ignite/table
'Table' is the primary API, which works with raw tuples. There are also
multiple views on top of it, including KeyValueView and
Ivan, what are extra serde steps you are talking about?
Best Regards,
Igor
On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky wrote:
> > I agree. But this was decided before in IEP-54, and is out of scope for
> current IEP.
> Would you like to start a separate thread to discuss this? Or I can do
> I agree. But this was decided before in IEP-54, and is out of scope for
current IEP.
Would you like to start a separate thread to discuss this? Or I can do this
a bit later.
Great idea, let's discuss it. I suppose this will simplify many aspects of
realization and improve performance a lot
чт,
> Here is the description of TUPLE_GET_ALL:
- UUID: table ID
- int: schema ID
- arr of arr: array of rows with values for all columns in given schema
I suppose that we should describe this more verbose and explicit. I
nevertheless suggest to also consider writing values this way:
- arr of fields
Guys, I've created the following ticket:
https://issues.apache.org/jira/browse/IGNITE-15046
Please feel free to comment or edit it.
If there are no objections, I'll create a PR soon.
Regards,
Nikita
чт, 1 июл. 2021 г. в 16:54, Nikita Safonov :
> Ivan, thank you for the links!
>
> Well, I can
Hi, Pavel.
I think, it looks like blocker. Please cherry-pick it to 2.11 release branch
On 2021/07/01 09:29:57, Pavel Pereslegin wrote:
> Hello, Alexey!
>
> Is it possible to include a hotfix that corrects the command syntax
> output in the control script? [1]
>
> This bug can
Hello Ivan,
> At least, we could just hide params that match a specific pattern
Yes, we can filter out all vm options that do not relate to Ignite, for
instance.
> Ilya, go ahead, file ticket and prepare a PR.
Please do not rush. Let's listen to other community members. This question
is about
> No it isn't, I have carefully read code and IEP, in your code you write
> schema id in each tuple.
There is no code for batch operations yet.
Here is the description of TUPLE_GET_ALL:
- UUID: table ID
- int: schema ID
- arr of arr: array of rows with values for all columns in given schema
(nil
AFAIK, this allows users to pass in custom VM options which may be undesirable.
On Thu, Jul 1, 2021 at 12:07 PM Zhenya Stanilovsky
wrote:
>
>
> +1 for reverting, can anybody (possibly ticket starter?) explain how jvm
> settings will rise some security problems ?
>
>
>
> >I suppose, that we
I suppose, that all normal users should not suffer from this restrictions.
Nobody will pass password using jvm options. It is absolutely insane,
normal users pass passwords using environment variables.
At least, we could just hide params that match specific pattern
Ilya, go ahead, file ticket
Ivan, thank you for the links!
Well, I can still find the docs on Python Thin Client under the "Thin
Сlients"->"Python Thin Client" in 2.10:
https://ignite.apache.org/docs/2.10.0/thin-clients/python-thin-client
So, I can replace the contents of "Thin Сlients"->"Python Thin Client"
page with a
Hello,
Unfortunately, the user can pass its own system properties via JVM options
as follows: -DMY_SECRET_PASSWORD=123
It does not seem, this approach is the best one. However, the user should
have a "kostyl" in order to hide these properties and values in the log
file, IMHO.
Thanks,
S.
ср, 30
Nikita,
What's wrong with platform-specific thin-client articles? These articles
describe how to configure and use specific thin client implementation, why
should we remove them?
чт, 1 июл. 2021 г. в 16:06, Ivan Daschinsky :
> Binary prorocol description should stand and improved (there are
> This is described in all operations that include multiple tuples.
No it isn't, I have carefully read code and IEP, in your code you write
schema id in each tuple.
Also, my biggest concern -- extra serde step. I suppose we should pass
bytearray to internal api, and use msgpack throughout all
Ivan,
> that there is not neccesary to write schema versions in each row
> in collectionof tuples
This is described in all operations that include multiple tuples.
> it is not clear from your code (probably
> mistake?) how differ key tuples and value tuples from each other
Key tuples include
Binary prorocol description should stand and improved (there are some
mistakes in this article)
чт, 1 июл. 2021 г., 16:05 Ivan Daschinsky :
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/
>
> AFAIK, thin clients have been already removed since 2.10
>
> чт, 1 июл. 2021
https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/
AFAIK, thin clients have been already removed since 2.10
чт, 1 июл. 2021 г., 15:57 Nikita Safonov :
> Hi guys!
>
> Sorry, totally missed this email.
>
> I can do this right away.
>
> Just a couple of questions first:
>
> [1]
Hi guys!
Sorry, totally missed this email.
I can do this right away.
Just a couple of questions first:
[1] On the website, under the Thin Clients tab, we have the following
articles:
Thin Clients
Thin Clients Overview
Java Thin Client
.NET Thin Client
C++ Thin Client
Python Thin Client
PHP
Hi.
We have made some progress on the topic.
The JRaft fork is merged to Ignite 3 master, now it's integrated with other
ready components.
The design of transactional protocol in the first iteration is published on
the master [1]
[1]
I suppose, that there is not neccesary to write schema versions in each row
in collectionof tuples. Also it is not clear from your code (probably
mistake?) how differ key tuples and value tuples from each other. In
readTuple you always read full schema and check for full length. As for me,
these
Ivan, tuple serialization section added to the IEP, let me know if it is
clear enough.
Thanks!
On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky wrote:
> I can't find any description of tuple serialization in IEP, only in code
>
> чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
>
> > Ivan,
> >
> > 0.
I can't find any description of tuple serialization in IEP, only in code
чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> Ivan,
>
> 0. The IEP is not in progress, it is ready for review and discussion.
> 1. Tuple serialization is described in the IEP and demonstrated in the PoC
> (see
Ivan,
0. The IEP is not in progress, it is ready for review and discussion.
1. Tuple serialization is described in the IEP and demonstrated in the PoC
(see ClientMessageHandler#readTuple), let me know if more details are
required
2. Tuple schema serialization is described in SCHEMAS_GET section.
Also, is there any clear information about KV api? Is there any plan to
implement it? Or is there any proposal about it?
чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> Pavel, but IEP is in progress, isn't it?
>
> 1. There is not any information about tuple serialization. And there isn't
> a clear
Pavel, but IEP is in progress, isn't it?
1. There is not any information about tuple serialization. And there isn't
a clear consensus about it.
2. There is not any information about schrma serialization format. And
AFAIK, there isn't a clear consensus also.
чт, 1 июл. 2021 г., 13:26 Pavel
Igniters,
Please review the IEP for thin client protocol in 3.0 [1].
PoC is in progress [2]
[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
[2] https://github.com/apache/ignite-3/pull/191
+1 (binding)
> 1 июля 2021 г., в 09:29, Petrov Mikhail написал(а):
>
> +1
>
> Checked on Ubuntu 20.04. Checked built-report and print-statistics scripts
> for the custom Ignite application. Checked archive signs. Built from the
> sources.
>
> On 01.07.2021 00:24, Sergei Ryzhov wrote:
>> +1
Hello, Alexey!
Is it possible to include a hotfix that corrects the command syntax
output in the control script? [1]
This bug can significantly complicate the use of the snapshot restore
function (one of the important features of 2.11). In addition, this
may raise a number of questions to the
Val, agreed.
Let's add length(), value(index), and Iterable to the Tuple interface.
I've updated the ticket: https://issues.apache.org/jira/browse/IGNITE-14342
On Wed, Jun 30, 2021 at 11:17 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:
> Pavel,
>
> Thanks for your response,
Hi, Iilya!
As I can see, this feature highly improves debugging during incidents. So I
think we can call it blocker and cherry-pick to ignite-2.11 branch
On 2021/06/30 20:26:43, Shishkov Ilya wrote:
> Hello, Alexey!
> Is it possible to add system views for BaselineNode attributes [1] and
>
I have opened a JIRA for the same:
https://issues.apache.org/jira/browse/IGNITE-15043
Unless objections, I plan on sketching an implementation plan.
On Thu, Jul 1, 2021 at 1:07 AM Atri Sharma wrote:
>
> Hi All,
>
> I have been playing around with Collision SPI and specifically used
>
+1 for reverting, can anybody (possibly ticket starter?) explain how jvm
settings will rise some security problems ?
>I suppose, that we should revert this particular line. I don't understand
>who ever considers vm options as sensitive info.
>
>ср, 30 июн. 2021 г., 22:52 Shishkov Ilya <
+1
Checked on Ubuntu 20.04. Checked built-report and print-statistics
scripts for the custom Ignite application. Checked archive signs. Built
from the sources.
On 01.07.2021 00:24, Sergei Ryzhov wrote:
+1
Checked a report with an ignite-performance-statistics-example.
I suppose, that we should revert this particular line. I don't understand
who ever considers vm options as sensitive info.
ср, 30 июн. 2021 г., 22:52 Shishkov Ilya :
> Hi Igniters,
>
> This feature [1, 2] prevents logging of the VM arguments when
> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is
56 matches
Mail list logo
