Re: IgniteFileSystem.create(...) Question--File Properties?
Hey, And is it required to have a file system APIs enabled? If not, you can deploy Ignite in a standard configuration (cache, data grid or IMDB) and use key-val, SQL, etc. for data access. - Denis On Wed, May 15, 2019 at 10:27 AM Chris Software wrote: > Hello! > > We are using it as a standalone, in order to create a distributed file > system for a (hopefully) fast cache of a few hours of data. We are not > using it in front of Hadoop, although there is some discussion of > eventually backing it with Cassandra, should we decide to keep more than a > few hours of data in the future. > > Chris > > On Tue, May 14, 2019 at 10:35 PM Denis Magda wrote: > >> Hello, >> >> Are you using IGFS as a standalone file system or as a way to accelerate >> Hadoop? If for the latter then I would suggest considering an alternate >> solution that is proved to be efficient for production deployment and >> Hadoop offloading: >> >> http://apache-ignite-users.70518.x6.nabble.com/SW-recommendation-Ignite-Native-Persistence-for-traditional-relational-data-warehouse-td28135.html >> >> - >> Denis >> >> >> On Tue, May 14, 2019 at 12:26 PM Chris Software < >> softwarechri...@gmail.com> wrote: >> >>> Hello, >>> >>> I am wondering what sort of File Properties can be set in ignite using >>> the IgniteFileSystem.create(IgfsPath path, int bufSize, boolean overwrite, >>> int replication, long, blockSize, Map props) method. >>> >>> I am using the default storage. It's not obvious to me in the code what >>> choices I have for this properties file. I don't see it documented in the >>> javadoc. >>> >>> Thank you. >>> >>
Re: SQL API in C++ thin client
Hi, It's not available yet, that's true. Hopefully, it will be added in the next few months. - Denis On Fri, May 17, 2019 at 6:24 AM matanlevy wrote: > Someone who can help? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Timestamp with Python thin client
Thank you very much. It is completely clear to me now. Kind regards, Stéphane Thibaud 2019年5月17日(金) 20:56 Igor Sapego : > These are nanosecond fraction of the last microsecond (It can be only from > 0 to 999). > > Ignite's Timestamp have a nanoseconds precision, so it can not be > represented > by the datetime type only. > > You can find some details here: [1] > > [1] - > https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/source/pyignite.datatypes.standard.html#pyignite.datatypes.standard.TimestampObject > > > Best Regards, > Igor > > > > On Thu, May 16, 2019 at 9:02 AM Stéphane Thibaud > wrote: > >> I see. Thank you. I am still a bit unsure about what the second value in >> the tuple represents. Are these indeed the nanoseconds? Apparently a Python >> datetime can have differing precisions (perhaps depending on platform)... >> >> >> Kind regards, >> >> Stéphane Thibaud >> >> >> 2019年5月16日(木) 0:10 Igor Sapego : >> >>> I believe, it's OK to pass tuple for timestamp in python, but you also >>> should >>> add a tip for the client to inform it you are going to store timestamp >>> value. >>> >>> Take a look at tests for example: [1] >>> >>> 1 - >>> https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L80 >>> >>> Best Regards, >>> Igor >>> >>> >>> On Wed, May 15, 2019 at 12:24 PM Stéphane Thibaud >>> wrote: >>> Thank you. I will try to create a code line to reproduce it, but I remember the following: when you do a 'select' query on a timestamp column with the Python thin client, you get a tuple. Because of that, I assumed that a tuple also had to be written in an update query. Kind regards, Stéphane Thibaud 2019年5月15日(水) 17:59 Igor Sapego : > Stéphane, > > Can you sharer a code line, how do you try to store timestamp value? > > Best Regards, > Igor > > > On Tue, May 14, 2019 at 7:23 PM Denis Mekhanikov < > dmekhani...@gmail.com> wrote: > >> Stéphane, >> >> Could you provide the code, that results in this exception? >> Do you try to insert the tuple as a single field via SQL? There is no >> such primitive as a tuple in SQL, so you should probably split timestamp >> into datetime and nanoseconds columns and store them separately as two >> different columns. >> >> Denis >> >> сб, 4 мая 2019 г. в 10:39, Stéphane Thibaud : >> >>> Hello Apache users, >>> >>> I am running into the following issue: when I try to store a >>> timestamp with nanosecond precision with the Python Thin client, I get >>> the >>> stack trace below. I have specified the timestamp as a tuple of >>> (datetime, >>> nanoseconds) as that is the format in which I also get timestamps back >>> from >>> the apache ignite client. Strangely, I can set just a datetime, but then >>> the nanoseconds become zero. Am I doing it in the wrong way? Any >>> suggestions? >>> >>> >>> >>> db.sql(query, query_args=[converted_row[c] for c in >>> table.column_names]) >>> File >>> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/client.py", >>> line 401, in sql >>> max_rows, timeout, >>> File >>> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/api/sql.py", >>> line 370, in sql_fields >>> 'include_field_names': include_field_names, >>> File >>> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/queries/__init__.py", >>> line 260, in from_python >>> buffer += c_type.from_python(values[name]) >>> File >>> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/datatypes/internal.py", >>> line 471, in from_python >>> buffer += infer_from_python(x) >>> File >>> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/datatypes/internal.py", >>> line 399, in infer_from_python >>> if is_hinted(value): >>> File >>> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/utils.py", >>> line 51, in is_hinted >>> and issubclass(value[1], IgniteDataType) >>> File "/usr/lib/python3.7/abc.py", line 143, in __subclasscheck__ >>> return _abc_subclasscheck(cls, subclass) >>> TypeError: issubclass() arg 1 must be a class >>> >>> Kind regards, >>> >>> Stéphane Thibaud >>> >>> >>>
Re: Need help with Linux ODBC Driver
Charles,
Thanks for the reproducer, I'll check it out.
Best Regards,
Igor
On Thu, May 16, 2019 at 6:59 PM Charles Rene
wrote:
> Thanks Igor. I also noticed that the Linux ODBC Driver deserializes the
> Ignite UUID data type into a byte[] instead of the .Net Guid type. That is
> contrary to what is stated in the documentation (
> https://apacheignite-sql.readme.io/docs/data-types#section-uuid).
>
> SQL Example:
> CREATE TABLE MyTable (Id INT not null, myUUID UUID, PRIMARY KEY (Id));
> INSERT INTO MyTable (Id, myUUID) VALUES (0,
> '24E4A97A-96DC-47EB-89C0-9C44E60DC048');
> SELECT myUUID FROM MyTable;
>
> C#:
> using System;
> using System.Data.Odbc;
>
> namespace IgniteUUIDWrongDataTypeReproducer {
> internal class Program {
> private static void Main(string[] args) {
> var connectionString =
> Environment.GetEnvironmentVariable("IGNITE_CONNECTION_STRING");
>
> using (var conn = new OdbcConnection(connectionString)) {
> conn.Open();
>
> using (var cmd = conn.CreateCommand()) {
> cmd.CommandText = "DROP TABLE IF EXISTS MyTable;";
> cmd.ExecuteNonQuery();
> }
>
> using (var cmd = conn.CreateCommand()) {
> cmd.CommandText = "CREATE TABLE MyTable (Id INT not
> null, myUUID UUID, PRIMARY KEY (Id));";
> cmd.ExecuteNonQuery();
> }
>
> using (var cmd = conn.CreateCommand()) {
> cmd.CommandText = "INSERT INTO MyTable (Id, myUUID)
> VALUES (0, '24E4A97A-96DC-47EB-89C0-9C44E60DC048');";
> cmd.ExecuteNonQuery();
> }
>
> using (var cmd = conn.CreateCommand()) {
> cmd.CommandText = "SELECT myUUID FROM MyTable;";
> using (var rdr = cmd.ExecuteReader()) {
> rdr.Read();
> var value = rdr[0];
> Console.WriteLine($"{rdr.GetName(0)}='{value}'
> DotnetType='{value.GetType()}' DBType='{rdr.GetDataTypeName(0)}'");
> }
> }
> }
> }
> }
> }
>
> Console Output:
> MYUUID='System.Byte[]' DotnetType='System.Byte[]' DBType='VARBINARY'
>
> Should be: (According to
> https://docs.microsoft.com/en-us/sql/odbc/reference/appendixes/sql-data-types?view=sql-server-2017
> )
> MYUUID='24E4A97A-96DC-47EB-89C0-9C44E60DC048' DotnetType='System.Guid'
> DBType='GUID'
>
>
>
>
>
> On Mon, May 13, 2019 at 9:55 AM Igor Sapego wrote:
>
>> It seems like some encoding related issue to me.
>> Added a Jira ticket: [1].
>>
>> [1] - https://issues.apache.org/jira/browse/IGNITE-11845
>>
>> Best Regards,
>> Igor
>>
>>
>> On Thu, May 2, 2019 at 1:15 AM Charles Rene
>> wrote:
>>
>>> Hello,
>>>
>>> I'm having trouble using the Apache Ignite 2.7.0 Linux built ODBC
>>> driver. I'm running a .Net Core 2.2 console application in a Linux
>>> environment in Docker.
>>>
>>> The problem is that when I run a SELECT statement through .Net's
>>> System.Data.Odbc, I get back "bad data". It looks like a mismatch in
>>> encoding.
>>>
>>> The data looks like "8\00\05\07\09\0d\09\08\0-\0" where as it should
>>> be "80579d98-9010-4610-b12e-ed33ed7d3c62".
>>>
>>> Details about my investigation can be found here:
>>>
>>> https://stackoverflow.com/questions/55875927/apache-ignite-2-7-odbc-linux-bad-data-returned
>>>
>>>
>>> Any idea what the problem might be?
>>>
>>> Thank you,
>>> Charlie
>>>
>>
Re: C++ Thin Client Lacks PC File
Is there anything I can do now to patch this now before its fixed upstream, or any idea on what is needed to patch this, I'd be willing to put in a PR for the fix, but I have mostly come up empty handed when trying to fix this locally. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: C++ Thin Client Lacks PC File
Hi, Sorry for not replying from the start. This is not intentional, and, I believe, should be fixed by the next release. Best Regards, Igor On Fri, May 17, 2019 at 4:12 PM mwilliamso58 wrote: > bump > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: SQL API in C++ thin client
Someone who can help? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: C++ Thin Client Lacks PC File
bump -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Timestamp with Python thin client
These are nanosecond fraction of the last microsecond (It can be only from 0 to 999). Ignite's Timestamp have a nanoseconds precision, so it can not be represented by the datetime type only. You can find some details here: [1] [1] - https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/source/pyignite.datatypes.standard.html#pyignite.datatypes.standard.TimestampObject Best Regards, Igor On Thu, May 16, 2019 at 9:02 AM Stéphane Thibaud wrote: > I see. Thank you. I am still a bit unsure about what the second value in > the tuple represents. Are these indeed the nanoseconds? Apparently a Python > datetime can have differing precisions (perhaps depending on platform)... > > > Kind regards, > > Stéphane Thibaud > > > 2019年5月16日(木) 0:10 Igor Sapego : > >> I believe, it's OK to pass tuple for timestamp in python, but you also >> should >> add a tip for the client to inform it you are going to store timestamp >> value. >> >> Take a look at tests for example: [1] >> >> 1 - >> https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L80 >> >> Best Regards, >> Igor >> >> >> On Wed, May 15, 2019 at 12:24 PM Stéphane Thibaud >> wrote: >> >>> Thank you. I will try to create a code line to reproduce it, but I >>> remember the following: when you do a 'select' query on a timestamp column >>> with the Python thin client, you get a tuple. Because of that, I assumed >>> that a tuple also had to be written in an update query. >>> >>> >>> Kind regards, >>> >>> Stéphane Thibaud >>> >>> 2019年5月15日(水) 17:59 Igor Sapego : >>> Stéphane, Can you sharer a code line, how do you try to store timestamp value? Best Regards, Igor On Tue, May 14, 2019 at 7:23 PM Denis Mekhanikov wrote: > Stéphane, > > Could you provide the code, that results in this exception? > Do you try to insert the tuple as a single field via SQL? There is no > such primitive as a tuple in SQL, so you should probably split timestamp > into datetime and nanoseconds columns and store them separately as two > different columns. > > Denis > > сб, 4 мая 2019 г. в 10:39, Stéphane Thibaud : > >> Hello Apache users, >> >> I am running into the following issue: when I try to store a >> timestamp with nanosecond precision with the Python Thin client, I get >> the >> stack trace below. I have specified the timestamp as a tuple of >> (datetime, >> nanoseconds) as that is the format in which I also get timestamps back >> from >> the apache ignite client. Strangely, I can set just a datetime, but then >> the nanoseconds become zero. Am I doing it in the wrong way? Any >> suggestions? >> >> >> >> db.sql(query, query_args=[converted_row[c] for c in >> table.column_names]) >> File >> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/client.py", >> line 401, in sql >> max_rows, timeout, >> File >> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/api/sql.py", >> line 370, in sql_fields >> 'include_field_names': include_field_names, >> File >> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/queries/__init__.py", >> line 260, in from_python >> buffer += c_type.from_python(values[name]) >> File >> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/datatypes/internal.py", >> line 471, in from_python >> buffer += infer_from_python(x) >> File >> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/datatypes/internal.py", >> line 399, in infer_from_python >> if is_hinted(value): >> File >> "/home/snthibaud/PycharmProjects/tabee/venv/lib/python3.7/site-packages/pyignite/utils.py", >> line 51, in is_hinted >> and issubclass(value[1], IgniteDataType) >> File "/usr/lib/python3.7/abc.py", line 143, in __subclasscheck__ >> return _abc_subclasscheck(cls, subclass) >> TypeError: issubclass() arg 1 must be a class >> >> Kind regards, >> >> Stéphane Thibaud >> >> >>
Re: auto scaling of ignite pods
Hi, I would recommend looking at the common approaches for pods scaling like Horizontal Pod Autoscaler: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/. Best Regards, Roman -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Integrity of write behind
John, Entries are queued for persisting only on primary nodes, so if it fails before writing all updates to the underlying database, then it will result in some entries not being written to the database at all. This is the price for a better performance, that write behind provides. Take a look at the following thread for more information: http://apache-ignite-users.70518.x6.nabble.com/Data-lost-when-using-write-behind-td4265.html Denis пт, 17 мая 2019 г. в 06:02, Coleman, JohnSteven (Agoda) < johnsteven.cole...@agoda.com>: > Hi, > > > > What happens if a node goes down while write behind is in progress on a > cache that’s persisting to the database? Can the task of persistence be > carried on exactly where it failed by a back up node? Are cache entries > flagged when they have been successfully persisted so that another node can > pick up the task later? Should the persistence layer keep a version number > or similar so that updates are orderly and not duplicated? > > > > John > > -- > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by copyright > or other legal rules. If you have received it by mistake please let us know > by reply email and delete it from your system. It is prohibited to copy > this message or disclose its content to anyone. Any confidentiality or > privilege is not waived or lost by any mistaken delivery or unauthorized > disclosure of the message. All messages sent to and from Agoda may be > monitored to ensure compliance with company policies, to protect the > company's interests and to remove potential malware. Electronic messages > may be intercepted, amended, lost or deleted, or contain viruses. >
Re: Store raw binary value in Apache Ignite through thin python client
Hi. I think the issue is related to building bytes counterpart of an array during serialization using += operation on bytes object which is immutable. This leads to throwing away previously created bytes instances, each of which grows at new iteration. This aligns with what I observe when run the test provided here. Quick fix - https://github.com/YevIgn/ignite/commit/64211b2d7c9e174f0c8979cbb4f26df52b8ebcd8 After it, the array building stops hanging on my machine. Should I open PR? Note: First saw the issue, the message here is copy from my comment in it. Best regards, Evgeniy Ignatiev. On 5/16/2019 4:42 PM, kulinskyvs wrote: Hi, Thank you for your feedback! Just for information - I've also created a stackoverflow question (https://stackoverflow.com/questions/56127565/store-raw-binary-value-in-apache-ignite-through-thin-python-client) related to this topic > I'll post the JIRA issue link there. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
