Re: [Maria-developers] b3844107287: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi Sergei,

On Tue, Feb 15, 2022 at 4:35 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> I've took a look at the whole diff bad1440325ba 9af2883e2692.
> (but without MDEV-16029)
>
> I have a couple of comments about minor issues, and a suggestion,
> please, see below. Nothing big, really.
>
> > diff --git a/include/my_base.h b/include/my_base.h
> > index 053bf3fbb69..f1c0d8dbc99 100644
> > --- a/include/my_base.h
> > +++ b/include/my_base.h
> > @@ -527,7 +527,8 @@ enum ha_base_keytype {
> >  #define HA_ERR_TABLESPACE_MISSING 194  /* Missing Tablespace */
> >  #define HA_ERR_SEQUENCE_INVALID_DATA 195
> >  #define HA_ERR_SEQUENCE_RUN_OUT   196
> > -#define HA_ERR_LAST   196  /* Copy of last error nr * */
> > +#define HA_ERR_WRONG_ROW_TIMESTAMP  197  /* System Versioning row_end 
> > <= row_start */
> > +#define HA_ERR_LAST   197  /* Copy of last error nr * */
>
> As you like. But I'd _suggest_ not to introduce a new HA_ERR_ code for this.
>
> You only use it in sql/, it's not something that a storage engine
> will use, as far as I understand. So there is no need to pollute the API,
> I would say.

Replaced with ER_WRONG_VALUE.

>
> >
> >  /* Number of different errors */
> >  #define HA_ERR_ERRORS(HA_ERR_LAST - HA_ERR_FIRST + 1)
> > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result 
> > b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result
> > index b28f3c567ff..6c09e0a7f23 100644
> > --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result
> > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result
> > @@ -982,6 +982,16 @@ NUMERIC_BLOCK_SIZE   1
> >  ENUM_VALUE_LIST  NULL
> >  READ_ONLYNO
> >  COMMAND_LINE_ARGUMENTREQUIRED
> > +VARIABLE_NAMEFORCE_FIELDS_VISIBLE
> > +VARIABLE_SCOPE   SESSION
> > +VARIABLE_TYPEBOOLEAN
> > +VARIABLE_COMMENT Make invisible fields visible on next table open
> > +NUMERIC_MIN_VALUENULL
> > +NUMERIC_MAX_VALUENULL
> > +NUMERIC_BLOCK_SIZE   NULL
> > +ENUM_VALUE_LIST  OFF,ON
> > +READ_ONLYNO
> > +COMMAND_LINE_ARGUMENTOPTIONAL
>
> apparently, forgot to update embedded result after renaming the variable

Fixed.

>
> >  VARIABLE_NAMEFOREIGN_KEY_CHECKS
> >  VARIABLE_SCOPE   SESSION
> >  VARIABLE_TYPEBOOLEAN
> > diff --git a/sql/log_event.h b/sql/log_event.h
> > index 3adc7a26d93..dc269955c5f 100644
> > --- a/sql/log_event.h
> > +++ b/sql/log_event.h
> > @@ -535,16 +535,12 @@ class String;
> >  */
> >  #define OPTIONS_WRITTEN_TO_BIN_LOG \
> >(OPTION_AUTO_IS_NULL | OPTION_NO_FOREIGN_KEY_CHECKS |  \
> > -   OPTION_RELAXED_UNIQUE_CHECKS | OPTION_NOT_AUTOCOMMIT | OPTION_IF_EXISTS)
> > +   OPTION_RELAXED_UNIQUE_CHECKS | OPTION_NOT_AUTOCOMMIT | OPTION_IF_EXISTS 
> > | \
> > +   OPTION_INSERT_HISTORY)
> >
> > -/* Shouldn't be defined before */
> > -#define EXPECTED_OPTIONS \
> > -  ((1ULL << 14) | (1ULL << 26) | (1ULL << 27) | (1ULL << 19) | (1ULL << 
> > 28))
> > -
> > -#if OPTIONS_WRITTEN_TO_BIN_LOG != EXPECTED_OPTIONS
> > -#error OPTIONS_WRITTEN_TO_BIN_LOG must NOT change their values!
> > +#if OPTIONS_WRITTEN_TO_BIN_LOG >= (1ULL << 32)
> > +#error OPTIONS_WRITTEN_TO_BIN_LOG must NOT exceed 32 bits!
>
> Well, no. This check makes sure that OPTION_RELAXED_UNIQUE_CHECKS
> is 1<<27, that OPTION_AUTO_IS_NULL is 1<<14, etc.

Reverted.

>
> For example, you've changed OPTION_SETUP_TABLES_DONE from bit 30 to 44.
> The EXPECTED_OPTIONS check was preventing somebody inadvertenly moving
> one of OPTIONS_WRITTEN_TO_BIN_LOG flags to a different bit.

And that helps to keep consistency between different versions of
master and slave, doesn't it? The above doesn't protect from the
swapped values between two flags.

>
> You can refactor it, if you don't like it.
> But please keep it in in some form.
>
> >  #endif
> > -#undef EXPECTED_OPTIONS /* You shouldn't use this one */
> >
> >  #define CHECKSUM_CRC32_SIGNATURE_LEN 4
> >  /**
> > diff --git a/sql/handler.cc b/sql/handler.cc
> > index 393f6234653..4bbb40abb5b 100644
> > --- a/sql/handler.cc
> > +++ b/sql/handler.cc
> > @@ -7493,6 +7496,26 @@ int handler::ha_write_row(const uchar *buf)
> >DBUG_RETURN(error);
> >}
> >
> > +  if (table->versioned() && !table->vers_write)
> > +  {
> > +Field *row_start= table->vers_start_field();
> > +Field *row_end= table->vers_end_field();
> > +MYSQL_TIME ltime;
> > +
> > +bitmap_set_bit(table->read_set, row_start->field_index);
> > +bitmap_set_bit(table->read_set, row_end->field_index);
> > +
> > +/*
> > +   Inserting the history row directly, check ROW_START <= ROW_END and
> > +   ROW_START is non-zero.
> > +*/
> > +if (!row_end->is_max() && (
> > +  (row_start->cmp(row_start->ptr, row_end->ptr) >= 0) ||
> > +  row_start->get_date(, Datetime::Options(
> > +TIME_NO_ZERO_DATE, 
> > time_round_mode_t(time_round_mode_t::FRAC_NONE)
>
> I don't quite understand 

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Aleksey Midenkov]

Sergei,

I updated the branch accordingly. There is your patch fa444975d0f
where I added log_current_statement().

On Wed, Apr 20, 2022 at 1:12 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> Okay, sure.
> I was going to push it after the release.
> And in 10.3 to simplify future merges.
>
> On Apr 20, Aleksey Midenkov wrote:
> > Hi Sergei!
> >
> > On Tue, Apr 19, 2022 at 11:13 AM Sergei Golubchik  wrote:
> > >
> > > Hi, Aleksey,
> > >
> > > On Apr 18, Aleksey Midenkov wrote:
> > > > >
> > > > > I have a second patch that renames flags to have a very well-defined
> > > > > meaning and name.
> > > > >
> > > > >   OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX
> > > > >   log_current_statement -> OPTION_BINLOG_THIS_STMT
> > > >
> > > > Where is your patch?
> > >
> > > It's really just renaming, almost nothing else.
> > > But here it is, attached.
> >
> > I'd keep log_current_statement() like this:
> >
> > --- a/sql/sql_class.h
> > +++ b/sql/sql_class.h
> > @@ -3562,4 +3562,8 @@ class THD: public THD_count, /* this must be first */
> >   /* set during loop of derived table processing */
> >   bool   derived_tables_processing;
> >   bool   tablespace_op;/* This is TRUE in DISCARD/IMPORT TABLESPACE 
> > */
> > +  bool   log_current_statement() const
> > +  {
> > +return variables.option_bits & OPTION_BINLOG_THIS_STMT;
> > +  }
> >   /**
> >
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Aleksey Midenkov]

Hi Sergei!

On Tue, Apr 19, 2022 at 11:13 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> On Apr 18, Aleksey Midenkov wrote:
> > >
> > > I have a second patch that renames flags to have a very well-defined
> > > meaning and name.
> > >
> > >   OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX
> > >   log_current_statement -> OPTION_BINLOG_THIS_STMT
> >
> > Where is your patch?
>
> It's really just renaming, almost nothing else.
> But here it is, attached.

I'd keep log_current_statement() like this:

--- a/sql/sql_class.h
+++ b/sql/sql_class.h
@@ -3562,4 +3562,8 @@ class THD: public THD_count, /* this must be first */
  /* set during loop of derived table processing */
  bool   derived_tables_processing;
  bool   tablespace_op;/* This is TRUE in DISCARD/IMPORT TABLESPACE */
+  bool   log_current_statement() const
+  {
+return variables.option_bits & OPTION_BINLOG_THIS_STMT;
+  }
  /**

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Aleksey Midenkov]

Hi Sergei,

On Sun, Apr 17, 2022 at 8:48 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> > I attached the patch and this still fails rpl.create_or_replace_row.
>
> small mistake in the patch:
>
> > --- a/sql/sql_class.h
> > +++ b/sql/sql_class.h
> > @@ -2916,7 +2916,7 @@ class THD: public THD_count, /* this must be first */
> >
> >bool binlog_need_stmt_format(bool is_transactional) const
> >{
> > -return vers_created_partitions && 
> > !binlog_get_pending_rows_event(is_transactional);
> > +return !binlog_get_pending_rows_event(is_transactional);
>
> if you put here
>
>   log_current_statement && 
> !binlog_get_pending_rows_event(is_transactional)
>
> then all tests pass.
>
> >}
> >
> >bool binlog_for_noop_dml(bool transactional_table);
>
> > Why don't you like vers_created_partitions? It might be useful info
> > for debugging and for new features.
>
> I have a second patch that renames flags to have a very well-defined
> meaning and name.
>
>   OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX
>   log_current_statement -> OPTION_BINLOG_THIS_STMT

Where is your patch?

>
> the first flag means that when a transaction is rolled back, it's
> written to binlog with ROLLBACK at the end, not discarded.
>
> The second means that the statement is written to binlog (or binlog trx
> cache) even if it otherwise wouldn't be.
>
> So, two flags. Clear names and semantics. "vers_created_partitions" fits
> into the second use case, so one should use OPTION_BINLOG_THIS_STMT for it.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Hi Sergei!

On Fri, Apr 15, 2022 at 6:00 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> On Apr 13, Aleksey Midenkov wrote:
> > >
> > > No, quite the opposite.
> > > I think (see above) that CONTEXT_ANALYSIS_ONLY_VCOL_EXPR
> > > (it's set inside init_lex_with_single_table()) is wrong, what you're
> > > doing is not "context analysys only", you're preparing items for
> > > evaluation.
> >
> > That works identically in branch and vanilla 10.3:
> >
> > --source include/have_ucs2.inc
> > create table t1 (c1 char(1) character set ucs2 collate ucs2_test_ci,
> >  v1 char(1) character set ucs2 collate ucs2_test_ci as (c1),
> >  v2 int as (c1 = 'b'),
> >  v3 int as (v1 = 'b'));
> > insert into t1 (c1) values ('a');
> > select * from t1 where v1 = 'b';
> > drop table t1;
> >
> > Result:
> > select * from t1 where v1 = 'b';
> > c1  v1  v2  v3
> > a   a   0   0
> >
> > .opt file:
> > --character-sets-dir=$MYSQL_TEST_DIR/std_data/ldml/
>
> Wow, this is a very good test. I wonder how you found it.

I disabled execution of Type_std_attributes::agg_item_set_converter()
and took the most appropriate failing test case.

>
> Yes, it shows, exactly, that it's incorrect to set
> CONTEXT_ANALYSIS_ONLY_VCOL_EXPR and then evaluate those items.
> Items fixed under CONTEXT_ANALYSIS_ONLY_VCOL_EXPR are *not fit* to be
> evaluated.
>
> Take a look at the bb-10.3-serg-MDEV-24176 branch.
> Three commits there: two cherry-picks from 10.2
> (they shouldn't be pushed into 10.3, instead your commit will eventually
> be rebased on top of them after they're merged into 10.3)
> and the "work-in-progress" commit that removes
> init_lex_with_single_table and CONTEXT_ANALYSIS_ONLY_VCOL_EXPR.

I just tried to note this is a different bug (as it reproduces in
master branch). And since it is lower
priority maybe we push the blocker first?

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Aleksey Midenkov]

Hi Sergei!

On Tue, Apr 12, 2022 at 12:58 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> Basically, it's fine, but I still didn't like that there's a new flag in
> THD to mark a statement that has to be logged. So I tried to change to
> log_current_statement. I know you tried that and you told me what tests
> fails, rpl.create_or_replace_*. I debugged that a bit, and I think the
> problem is that select_create inherits from select_insert, so
> CREATE ... SELECT runs your code that is only supposed to be run for
> INSERT ODKU or REPLACE. This change makes all tests pass (after removing
> vers_created_partitions):

I attached the patch and this still fails rpl.create_or_replace_row.
Why don't you like vers_created_partitions? It might be useful info
for debugging and for new features.

>
> @@ -4225,7 +4225,8 @@ bool select_insert::prepare_eof()
>thd->clear_error();
>  else
>errcode= query_error_code(thd, killed_status == NOT_KILLED);
> -StatementBinlog stmt_binlog(thd, 
> thd->binlog_need_stmt_format(trans_table));
> +StatementBinlog stmt_binlog(thd, !can_rollback_data() &&
> +thd->binlog_need_stmt_format(trans_table));
>  res= thd->binlog_query(THD::ROW_QUERY_TYPE,
> thd->query(), thd->query_length(),
>     trans_table, FALSE, FALSE, errcode);
>
> On Apr 10, Aleksey Midenkov wrote:
> > revision-id: 17548c8a8b6 (mariadb-10.6.1-250-g17548c8a8b6)
> > parent(s): 6282e025a0e
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2022-03-07 23:49:35 +0300
> > message:
> >
> > MDEV-25477 Auto-create breaks replication when triggering event was not 
> > replicated
> >
> > diff --git a/sql/sql_class.h b/sql/sql_class.h
> > index 686e6e70766..4fed1bf7355 100644
> > --- a/sql/sql_class.h
> > +++ b/sql/sql_class.h
> > @@ -2914,6 +2914,13 @@ class THD: public THD_count, /* this must be first */
> >int binlog_flush_pending_rows_event(bool stmt_end, bool 
> > is_transactional);
> >int binlog_remove_pending_rows_event(bool clear_maps, bool 
> > is_transactional);
> >
> > +  bool binlog_need_stmt_format(bool is_transactional) const
> > +  {
> > +return vers_created_partitions && 
> > !binlog_get_pending_rows_event(is_transactional);
> > +  }
> > +
> > +  bool binlog_for_noop_dml(bool transactional_table);
> > +
> >/**
> >  Determine the binlog format of the current statement.
> >
> > @@ -3557,6 +3564,7 @@ class THD: public THD_count, /* this must be first */
> >bool   tablespace_op;  /* This is TRUE in DISCARD/IMPORT TABLESPACE 
> > */
> >/* True if we have to log the current statement */
> >boollog_current_statement;
> > +  uint   vers_created_partitions;
> >/**
> >  True if a slave error. Causes the slave to stop. Not the same
> >  as the statement execution error (is_error()), since
> > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc
> > index 707b8a0d3bf..25551dfa197 100644
> > --- a/sql/sql_insert.cc
> > +++ b/sql/sql_insert.cc
> > @@ -4223,6 +4225,7 @@ bool select_insert::prepare_eof()
> >thd->clear_error();
> >  else
> >errcode= query_error_code(thd, killed_status == NOT_KILLED);
> > +StatementBinlog stmt_binlog(thd, 
> > thd->binlog_need_stmt_format(trans_table));
> >  res= thd->binlog_query(THD::ROW_QUERY_TYPE,
> > thd->query(), thd->query_length(),
> > trans_table, FALSE, FALSE, errcode);
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok
diff --git a/sql/partition_info.cc b/sql/partition_info.cc
index 3a4d690bd46..f8e8369fedc 100644
--- a/sql/partition_info.cc
+++ b/sql/partition_info.cc
@@ -1028,7 +1028,6 @@ bool vers_create_partitions(THD *thd, TABLE_LIST* tl, uint num_parts)
   thd->get_stmt_da()->reset_diagnostics_area();
   thd->variables.option_bits|= OPTION_KEEP_LOG;
   thd->log_current_statement= true;
-  thd->vers_created_partitions= num_parts;
 
 exit:
   thd->work_part_info= save_part_info;
diff --git a/sql/sql_class.h b/sql/sql_class.h
index 4fed1bf7355..29bcfe98b08 100644
--- a/sql/sql_class.h
+++ b/sql/sql_class.h
@@ -2916,7 +2916,7 @@ class THD: public THD_count, /* this must be first */
 
   bool binlog_need_stmt_format(bool is_transactional) const
   {
-return vers_created_partitions && !binlog_get_pending_rows_event(is_transactional);
+return !binlog_get_pending_rows_event(i

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Hi Sergei,

On Wed, Apr 13, 2022 at 8:18 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> On Apr 12, Aleksey Midenkov wrote:
> > > > > > +bool Vcol_expr_context::init()
> > > > > > +{
> > > > > > +  /*
> > > > > > +  As this is vcol expression we must narrow down name 
> > > > > > resolution to
> > > > > > +  single table.
> > > > > > +  */
> > > > > > +  if (init_lex_with_single_table(thd, table, ))
> > > > >
> > > > CURRENT_TEST: gcol.gcol_bugfixes
> > > > mysqltest: At line 579: query 'INSERT INTO t1 (suppliersenttoday)
> > > > VALUES (0)' failed: 2013: Lost connection to MySQL server during query
> > >
> > > this one crashes. on the next line, 580, though.
> > > because you set CONTEXT_ANALYSIS_ONLY_VCOL_EXPR, so
> > > Type_std_attributes::agg_item_set_converter does not wrap items
> > > in Item_func_conv_charset. Which is likely incorrect, because items
> > > without wrapping cannot be properly evaluated, and it looks like they
> > > has to be evaluated later, so it's not "context analysys only".
> > >
> > > Crash on wrapped items is https://jira.mariadb.org/browse/MDEV-25638
> > > that Sanja is looking at right now.
> >
> > Agree, I have the same picture now. Previous faults were on work in
> > progress. So it faults anyway and we keep
> > init_lex_with_single_table(), right?
>
> No, quite the opposite.
> I think (see above) that CONTEXT_ANALYSIS_ONLY_VCOL_EXPR
> (it's set inside init_lex_with_single_table()) is wrong, what you're
> doing is not "context analysys only", you're preparing items for
> evaluation.

That works identically in branch and vanilla 10.3:

--source include/have_ucs2.inc
create table t1 (c1 char(1) character set ucs2 collate ucs2_test_ci,
 v1 char(1) character set ucs2 collate ucs2_test_ci as (c1),
 v2 int as (c1 = 'b'),
 v3 int as (v1 = 'b'));
insert into t1 (c1) values ('a');
select * from t1 where v1 = 'b';
drop table t1;

Result:
select * from t1 where v1 = 'b';
c1  v1  v2  v3
a   a   0   0

.opt file:
--character-sets-dir=$MYSQL_TEST_DIR/std_data/ldml/

Please, correct the test case if I missed something.
...

As a side note here I tried to clear CONTEXT_ANALYSIS_ONLY_VCOL_EXPR:
https://github.com/midenok/mariadb/issues/93#issuecomment-1098336828

Couple of tests failed.

>
> Let's wait for Sanja to close his MDEV-25638, and then I'll check this
> test case again.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Sergei,

On Wed, Apr 6, 2022 at 11:03 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> Thanks. I'll send a separate review email, there will be only replies
> here:
>
> On Apr 05, Aleksey Midenkov wrote:
> > Hi, Sergei!
> >
> > > > @@ -5709,8 +5708,6 @@ bool Item_field::fix_fields(THD *thd, Item 
> > > > **reference)
> > > >}
> > > >  #endif
> > > >fixed= 1;
> > > > -  if (field->vcol_info)
> > > > -fix_session_vcol_expr_for_read(thd, field, field->vcol_info);
> > >
> > > Did you revert the optimization from 7450cb7f69db?
> > > It was "re-fix vcols on demand, not always for every SELECT"
> > >
> > > and as far as I can see now you again always re-fix everything,
> > > is that so?
> >
> > It was your suggestion in the previous review to refix always
> > (therefore everything). The original fix worked with
> > fix_session_vcol_expr_for_read(). But the current fix is simpler and
> > cleaner. Was there a real performance problem or was it a theoretical
> > issue?
>
> I checked previous emails and wasn't able to find where I suggested to
> refix always.

Sorry, you suggested to cleanup in the end of statement and I came to
fix always in the implementation. That of course different things.

>
> Anyway, it was indeed a theoretical issue and I have no proofs to claim
> that that "optimization" made a difference. So, okay, feel free to refix
> always.
>
> > > >if (thd->variables.sql_mode & MODE_ONLY_FULL_GROUP_BY &&
> > > >!outer_fixed &&
> > > >select &&
> > > > diff --git a/sql/table.h b/sql/table.h
> > > > index 38b63d285c6..ab1d96538c0 100644
> > > > --- a/sql/table.h
> > > > +++ b/sql/table.h
> > > > @@ -1374,8 +1374,13 @@ struct TABLE
> > > >*/
> > > >bool auto_increment_field_not_null;
> > > >bool insert_or_update; /* Can be used by the handler */
> > > > +  /*
> > > > + NOTE: alias_name_used is only a hint! It works only in 
> > > > need_correct_ident()
> > > > + condition. On other cases it is FALSE even if table_name is alias!
> > >
> > > As I asked earlier, do you have any test case that proves this claim?
> >
> > I replied that MDEV-25672 bug's own test case proves that, didn't I?
> >
> > (rr) p alias_name_used
> > $4 = false
> > (rr) p table_name
> > $5 = 0x7fdc28024790 "x"
> >
> > #0  Item_field::set_field (this=0x7fdc28012e60,
> > field_par=0x7fdc28024440) at /home/midenok/src/mariadb/10
> > .3/src/sql/item.cc:3289
> > #1  0x00b7bc28 in Item_field::fix_fields (this=0x7fdc28012e60,
> > thd=0x7fdc28000d28, reference=0x7fdc28034ad8) at 
> > /home/midenok/src/mariadb/10.3/src/sql/item.cc:6332
> > ...
> > #15 0x007c65f6 in mysql_parse (thd=0x7fdc28000d28,
> > rawbuf=0x7fdc28010520 "UPDATE `test_table` as
> > `x` SET order_date = NULL", length=48, parser_state=0x7fdc39d09548,
> > is_com_multi=false, is_next_command=false) at 
> > /home/midenok/src/mariadb/10.3/src/sql/sql_parse.cc:7870
>
> indeed, thanks. Could you add it to the comment? Like "e.g. in update t1 as x 
> set a = 1"

Added.

>
> > > > +  */
> > > >bool alias_name_used;  /* true if table_name is alias */
> > > >bool get_fields_in_item_tree;  /* Signal to fix_field */
> > > > +  List vcol_cleanup_list;
> > > >CHARSET_INFO *save_character_set_client= 
> > > > thd->variables.character_set_client;
> > > >CHARSET_INFO *save_collation= thd->variables.collation_connection;
> > > >Query_arena  *backup_stmt_arena_ptr= thd->stmt_arena;
> > > > @@ -2831,36 +2841,162 @@ static bool fix_vcol_expr(THD *thd, 
> > > > Virtual_column_info *vcol)
> ...
> > > > +bool Vcol_expr_context::init()
> > > > +{
> > > > +  /*
> > > > +  As this is vcol expression we must narrow down name resolution to
> > > > +  single table.
> > > > +  */
> > > > +  if (init_lex_with_single_table(thd, table, ))
> > >
> > > Do you have a test that fails otherwise?
> >
> > That fails at least 3 tests:
> >
> > main.default vcol.vcol_syntax gcol.gcol_bugfixes
> >
> > CURRENT_TEST: main.default
> > mysqltest: At line 1302: query 'INSERT INTO t1 VALUES( DEFAULT

Re: [Maria-developers] 62d3496969d: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Hi Sergei!

On Thu, Apr 7, 2022 at 8:09 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> This is a review of `git diff 5fbbdbc85ba3 62d3496969db`
>
> Basically, no new comments, see below:
>
> On Apr 07, Aleksey Midenkov wrote:
>
> > diff --git a/libmariadb b/libmariadb
> > index f6c3d9fd2af..735a7299dba 16
> > --- a/libmariadb
> > +++ b/libmariadb
> > @@ -1 +1 @@
> > -Subproject commit f6c3d9fd2af5d17db64cc996574aa312efd70fcf
> > +Subproject commit 735a7299dbae19cc2b82b9697becaf90e9b43047
>
> you've mistakenly checked in a submodule

Removed that.

>
> > diff --git a/sql/field.cc b/sql/field.cc
> > index f18fb25ebe3..04c492e6c69 100644
> > --- a/sql/field.cc
> > +++ b/sql/field.cc
> > @@ -2416,6 +2416,13 @@ int Field::set_default()
> >if (default_value)
> >{
> >  Query_arena backup_arena;
> > +/*
> > +  TODO: this imposes memory leak until table flush when save_in_field()
> > +does expr_arena allocation. F.ex. case from main.default:
> > +
> > +CREATE TABLE t1 (a INT DEFAULT CONCAT('1 '));
> > +INSERT INTO t1 VALUES (DEFAULT);
>
> I wasn't able to repeat that. In your branch (didn't try in vanilla 10.3) and
> for the above test case I see no additional allocations in table->mem_root
> after the table is opened.

Right, that was a different mem_root. I cannot find the test case for
that, though I found test cases for update_virtual_fields(). I updated
the comments. The several stack traces and the patch for catching
mem_root usage can be found here:

https://gist.github.com/midenok/fe9ef9de39ff5ed0f89e4e344db76a91

>
> > +*/
> >  table->in_use->set_n_backup_active_arena(table->expr_arena, 
> > _arena);
> >  int rc= default_value->expr->save_in_field(this, 0);
> >  table->in_use->restore_active_arena(table->expr_arena, _arena);
> > diff --git a/sql/table.cc b/sql/table.cc
> > index 08d91678b25..d22da17c31a 100644
> > --- a/sql/table.cc
> > +++ b/sql/table.cc
> > @@ -2970,36 +2988,133 @@ static bool fix_vcol_expr(THD *thd, 
> > Virtual_column_info *vcol)
> ...
> > +bool Vcol_expr_context::init()
> > +{
> > +  /*
> > +  As this is vcol expression we must narrow down name resolution to
> > +  single table.
> > +  */
> > +  if (init_lex_with_single_table(thd, table, ))
>
> I've already asked about it in a separate email,
> so here I'm just marking a line that we haven't fully agreed on yet

You are right, I have the same picture now. I tried on some different
(intermediate) revision, so that was the different result.

>
> > +  {
> > +my_error(ER_OUT_OF_RESOURCES, MYF(0));
> > +table->map= old_map;
> > +return true;
> > +  }
> > +
> > +  lex.sql_command= old_lex->sql_command;
> > +  thd->variables.sql_mode= 0;
> > +
> > +  TABLE_LIST const *tl= table->pos_in_table_list;
> > +  DBUG_ASSERT(table->pos_in_table_list);
> > +
> > +  if (table->pos_in_table_list->security_ctx)
> >  thd->security_ctx= tl->security_ctx;
> > -  bool res= fix_session_vcol_expr(thd, vcol);
> > +
> > +  inited= true;
> > +  return false;
> > +}
> > +
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Hi, Sergei!

On Thu, Mar 31, 2022 at 11:30 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> Here's a review of the combined `git diff 4daf9d7c3ee0 f67dcf6ba5f2`:
>
> > diff --git a/mysql-test/suite/vcol/r/vcol_syntax.result 
> > b/mysql-test/suite/vcol/r/vcol_syntax.result
> > index 0063f38ea36..c5c8a33c0ec 100644
> > --- a/mysql-test/suite/vcol/r/vcol_syntax.result
> > +++ b/mysql-test/suite/vcol/r/vcol_syntax.result
> > @@ -94,6 +94,68 @@ create table t1 (a int, v_a int generated always as (a));
> >  update t1 as x set a = 1;
> >  alter table t1 force;
> >  drop table t1;
> > +create table t1 (
> > +id int not null auto_increment primary key,
> > +order_date_time datetime not null,
> > +order_date date generated always as (convert(order_date_time, date)),
> > +language_id binary(16) null
> > +);
> > +update t1 as tx set order_date= null;
> > +alter table t1 modify column language_id binary(16) not null;
> > +drop table t1;
> >  #
> > -# End of 10.2 tests
> > +# MDEV-24176 Server crashes after insert in the table with virtual column 
> > generated using date_format() and if()
> >  #
> > +create table t1 (d1 date not null, d2 date not null,
> > +gd text as (concat(d1,if(d1 <> d2, date_format(d2, 'to %y-%m-%d '), ''))) 
> > );
> > +insert into t1(d1,d2) values
> > +('2020-09-01','2020-09-01'),('2020-05-01','2020-09-01');
>
> would be good to have a SELECT here, to verify the server not only
> not crashes, but actually inserts something

Added.

>
> > +drop table t1;
> > +# MDEV-25772 (duplicate) and LOCK TABLES case
> > +create table t1 (d1 datetime , v_d1 tinyint(1) as (d1 < curdate()));
> > +insert into t1 (d1) values ('2021-09-11 08:38:23'), ('2021-09-01 
> > 08:38:23');
> > +lock tables t1 write;
> > +select * from t1 where v_d1=1;
> > +d1   v_d1
> > +2021-09-11 08:38:23  1
> > +2021-09-01 08:38:23  1
> > +select * from t1;
> > +d1   v_d1
> > +2021-09-11 08:38:23  1
> > +2021-09-01 08:38:23  1
> > +unlock tables;
> > +drop table t1;
> > +# MDEV-26432 (duplicate)
> > +create table t1 (v2 int, v1 int as ((user() like 'x'))) ;
> > +select 1 from t1 where v1=1 ;
> > +1
> > +select * from t1;
> > +v2   v1
> > +drop table t1;
> > +create table t1 (v2 int as ( user () like 'x'));
> > +select 1 from t1 order by v2 ;
> > +1
> > +alter table t1 add i int;
> > +drop table t1;
> > +# MDEV-26437 (duplicate)
> > +create table v0 (v2 int not null,
> > +v1 bigint as (case 'x' when current_user() then v2 end));
> > +select v2 as v3 from v0 where v1 like 'x' escape 'x';
> > +v3
> > +insert into v0 (v2) values (-128);
> > +drop table v0;
> > +create table t1 (vi int as (case 'x' when current_user() then 1 end));
> > +select 1 from t1 where vi=1;
> > +1
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `vi` int(11) GENERATED ALWAYS AS (case 'x' when current_user() then 1 
> > end) VIRTUAL
> > +) ENGINE=MyISAM DEFAULT CHARSET=latin1
> > +drop table t1;
> > +create table t1 (vi int as (case 'x' when current_user() then 1 end));
> > +select 1 from t1 where vi=1;
> > +1
> > +select 1 from t1 where vi=1;
> > +1
> > +drop table t1;
> > diff --git a/sql/field.cc b/sql/field.cc
> > index 4e6bc6b8341..297edd9e75c 100644
> > --- a/sql/field.cc
> > +++ b/sql/field.cc
> > @@ -2453,6 +2453,7 @@ int Field::set_default()
> >if (default_value)
> >{
> >  Query_arena backup_arena;
> > +/* TODO: this imposes memory leak until table flush */
>
> could you add an example here? in what case, exactly, there
> can be a memory leak?

The change was introduced by db7edfed17ef
Updated the comment to:

/*
  TODO: this imposes memory leak until table flush when save_in_field()
does expr_arena allocation. F.ex. case from main.default:

CREATE TABLE t1 (a INT DEFAULT CONCAT('1 '));
INSERT INTO t1 VALUES (DEFAULT);
*/

>
> you have more comments like that below, I'd suggest to add one example
> somewhere, and change other comments to say "see ... for an example"

Ok, but the other code does allocations for other cases. I'm not sure
if it is needed to find all the test cases and why it is needed.

>
> >  table->in_use->set_n_backup_active_arena(table->expr_arena, 
> > _arena);
> >  int rc= default_value->expr->save_in_field(this, 0);
> >  table->in_use->restore_active_arena(table->expr_arena, _arena);
> > diff --git a/sql/item.h b/sql/item.h
> > index 2a904c1691a..e3028ae412c 100644
> > --- a/sql/item.h
> > +++ b/sql/item.h
> > @@ -2587,6 +2587,12 @@ class Item_ident :public Item_result_field
> >const char *db_name;
> >const char *table_name;
> >const char *field_name;
> > +  /*
> > + NOTE: came from TABLE::alias_name_used and this is only a hint! It 
> > works
> > + only in need_correct_ident() condition. On other cases it is FALSE 
> > even if
> > + table_name is alias! It cannot be TRUE in these cases, lots of 
> > spaghetti
> > + logic depends on that.
> > +  */
>
> this is a copy of the comment in 

Re: [Maria-developers] 88f8aa20bba: MDEV-23486 RBR can bypass secure_timestamp=YES

[Aleksey Midenkov]

Hi, Sergey!

On Sun, Jan 9, 2022 at 4:53 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jan 09, Aleksey Midenkov wrote:
> > revision-id: 88f8aa20bba (mariadb-10.3.31-79-g88f8aa20bba)
> > parent(s): d5451867af1
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-12-23 04:15:39 +0300
> > message:
> >
> > MDEV-23486 RBR can bypass secure_timestamp=YES
> >
> > Pass to a slave whether master table is system-versioned via
> > TM_SYSTEM_VERSIONED flag. That deprecates m_vers_from_plain
> > detection. If secure_timestamp == YES or master did not supply row_end
> > data treat the row as current data and generate timestamps for it.
>
> I don't understand why does it matter whether the master is system
> versioned or not.
>
> secure_timestamp=YES means the table has to store actual timestamps when
> rows were changed. It doesn't trust the master to set timestamps. So
> it should not matter whether master table is versioned or not, if you
> don't trust the master, you cannot trust its TM_SYSTEM_VERSIONED flag
> either. Even if the table was system versioned on the master, the master
> could've had secure_timestamp=NO and a completely fake history.

If slave ignores completely row_start/row_end coming from the master
how it can tell if row is history or not? UPDATE sends 2 events:
Update_rows and Write_rows. How can we tell what row_end value to
assign when processing Write_rows: as for history or as for current
data?

If the master can fake history it can fake any data. So if we even
somehow protected the history, like generate always on slave, if
master sends fake current data then it becomes fake history. I don't
see a good reason for such a kind of protection. The better protection
would be to organize secure communication: encrypt binlog, sign events
with certificates, etc.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 0402f6dcb67: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Aleksey Midenkov]

Hi Sergei,

On Wed, Mar 2, 2022 at 10:33 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> On Feb 15, Aleksey Midenkov wrote:
> > > On Feb 12, Aleksey Midenkov wrote:
> > > > commit 0402f6dcb67
> > > > Author: Aleksey Midenkov 
> > > > Date:   Tue Feb 8 22:54:03 2022 +0300
> > > >
> > > > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc
> > > > index 0403d6e73c3..06068290247 100644
> > > > --- a/sql/sql_delete.cc
> > > > +++ b/sql/sql_delete.cc
> > > > @@ -507,6 +508,18 @@ bool mysql_delete(THD *thd, TABLE_LIST 
> > > > *table_list, COND *conds,
> > > >  if (thd->lex->describe || thd->lex->analyze_stmt)
> > > >goto produce_explain_and_leave;
> > > >
> > > > +if (thd->log_current_statement)
> > > > +{
> > > > +  thd->reset_unsafe_warnings();
> > > > +  if (thd->binlog_query(THD::STMT_QUERY_TYPE,
> > > > +thd->query(), thd->query_length(),
> > > > +transactional_table, FALSE, FALSE, 0) > 0)
> > > > +  {
> > > > +my_error(ER_ERROR_ON_WRITE, MYF(0), "binary log", -1);
> > > > +DBUG_RETURN(1);
> > > > +  }
> > > > +}
> > >
> > > perhaps, move this block into a function? It's repeated four times here.
> > >
> >
> > Moved.
>
> Thanks!
> The new method, binlog_for_noop_dml() looks very puzzling.
>
> I understand what it does, for now, but somebody who wasn't involved in
> this bugfixing, likely won't have any idea.
> The name is correct, I agree, it says what, but it doesn't explain why.
>
> Could you add a comment there? To say that "noop dml" is "an
> insert(odku)/update/delete that doesn't change the table" and that
> it is normally not logged, but needs to be logged if it auto-created a
> partition as a side effect.

Added.

>
> > > by the way, you don't seem to have a test for a zero-rows insert.
> > > That is INSERT ... SELECT with an impossible where or limit 0.
> > >
> > > In fact, you can have a zero-row insert with a normal insert
> > > INSERT t1 VALUES (1); if this (1) is a unique key violation.
> > > I believe your code handles this case, thought, just a test case is
> > > missing.
> >
> > INSERT does not trigger history. But INSERT .. ODKU does. Added test
> > cases for INSERT .. ODKU and INSERT .. SELECT .. ODKU. That required
> > to add THD::vers_created_partitions as log_current_statement is also
> > used in CREATE TABLE (and select_insert::prepare_eof() is used for
> > CREATE TABLE .. SELECT) and that broke some replication tests:
> >
> > rpl.create_or_replace_mix
> > rpl.create_or_replace_row
> > rpl.create_or_replace_statement
>
> Indeed, I see.
>
> Can you use OPTION_KEEP_LOG instead?
> Or transaction->stmt.modified_non_trans_table ?

I don't think it is a good idea to intermix this semantic into these
variables/flags. log_current_statement was designed specifically for
this.

>
> I hate it that there're so many ways to force binlogging a statement,
> and they're all, of course, are subtly different.

That is pretty normal. It is the business logic that is complex and
therefore its realization should be complex too. Compressing many
semantics into single control variables only makes development harder.

>
> >
> > > > +
> > > >  my_ok(thd, 0);
> > > >  DBUG_RETURN(0);
> > > >}
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 0402f6dcb67: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Aleksey Midenkov]

Hi Sergei!

Updated branch preview-10.8-MDEV-17554-auto-create-partition

On Sat, Feb 12, 2022 at 2:31 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey,
>
> This was quite good, thanks!
>
> Many cases covered, not just ROLLBACK, but also LIMIT 0 and
> impossible WHERE.
>
> See a couple of comments below. Nothing big.
>
> On Feb 12, Aleksey Midenkov wrote:
> > commit 0402f6dcb67
> > Author: Aleksey Midenkov 
> > Date:   Tue Feb 8 22:54:03 2022 +0300
> >
> > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc
> > index 0403d6e73c3..06068290247 100644
> > --- a/sql/sql_delete.cc
> > +++ b/sql/sql_delete.cc
> > @@ -507,6 +508,18 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, 
> > COND *conds,
> >  if (thd->lex->describe || thd->lex->analyze_stmt)
> >goto produce_explain_and_leave;
> >
> > +if (thd->log_current_statement)
> > +{
> > +  thd->reset_unsafe_warnings();
> > +  if (thd->binlog_query(THD::STMT_QUERY_TYPE,
> > +thd->query(), thd->query_length(),
> > +transactional_table, FALSE, FALSE, 0) > 0)
> > +  {
> > +my_error(ER_ERROR_ON_WRITE, MYF(0), "binary log", -1);
> > +DBUG_RETURN(1);
> > +  }
> > +}
>
> perhaps, move this block into a function? It's repeated four times here.
>

Moved.

> by the way, you don't seem to have a test for a zero-rows insert.
> That is INSERT ... SELECT with an impossible where or limit 0.
>
> In fact, you can have a zero-row insert with a normal insert
> INSERT t1 VALUES (1); if this (1) is a unique key violation.
> I believe your code handles this case, thought, just a test case is missing.
>

INSERT does not trigger history. But INSERT .. ODKU does. Added test
cases for INSERT .. ODKU and INSERT .. SELECT .. ODKU. That required
to add THD::vers_created_partitions as log_current_statement is also
used in CREATE TABLE (and select_insert::prepare_eof() is used for
CREATE TABLE .. SELECT) and that broke some replication tests:

rpl.create_or_replace_mix
rpl.create_or_replace_row
rpl.create_or_replace_statement

> > +
> >  my_ok(thd, 0);
> >  DBUG_RETURN(0);
> >}
> > @@ -932,8 +958,8 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, 
> > COND *conds,
> >else
> >  errcode= query_error_code(thd, killed_status == NOT_KILLED);
> >
> > -  ScopedStatementReplication scoped_stmt_rpl(
> > -  table->versioned(VERS_TRX_ID) ? thd : NULL);
> > +  StatementBinlog stmt_binlog(thd, table->versioned(VERS_TRX_ID) ||
> > +   
> > thd->binlog_need_stmt_format(transactional_table));
>
> I know that's not part of this patch, but still. Just trying to understand.
> Why did it change to statement-based if VERS_TRX_ID?

Because we don't need replicated TRX_ID as they are different on
slave. The related commit is d998da03069 and the issue is
https://github.com/tempesta-tech/mariadb/issues/234


>
> >/*
> >  [binlog]: If 'handler::delete_all_rows()' was called and the
> >  storage engine does not inject the rows itself, we replicate
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



--
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Hi Sergei!

On Mon, Jan 17, 2022 at 7:56 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jan 13, Aleksey Midenkov wrote:
> > >
> > > But always using expr_arena doesn't necessarily seem to be a correct
> > > choice. E.g. Arg_comparator creates Item_cache_temporal on every
> > > fix_field. If vcol expr is re-fixed all the time, expr_arena will
> > > grow continuously. Same for charset conversion, CASE, IN, etc. I
> > > suspect that if the vcol expr is always re-fixed, it has to use stmt
> > > arena.
> >
> > stmt_arena cannot be used as long as 'expr' item is created on
> > expr_arena. 'expr' item is created at the vcol expression parse and is
> > reused during the whole TABLE lifetime. If some of the containee items
> > later were allocated on stmt_arena, 'expr' item will try to access
> > them in a different statement when they are already released. That is
> > what this bugfix tries to address.
> >
> > Yes, this fix introduces a small and in most cases imperceptible
> > memory leak which can be easily overcame by FLUSH TABLES. The fix for
> > this leak is non-trivial.
> >
> > 1. vcol expr is not always refixed, but conditionally controlled by
> > vcols_need_refixing. And we must remove this control and truly refix
> > always.
> >
> > 2. we must clone 'expr' item into statement mem_root at each
> > statement.
> >
> > I have attached the leak fix PoC into this email. I do not know
> > whether this is worth it.
>
> vcols_need_refixing isn't run-time conditional, it depends on vcol
> flags, a specific vcol always needs refixing or never does.
> And a table has vcols_need_refixing if any of the vcols nees it.
>
> Thus, I think, a simpler fix would be to split the cleanup and
> fix_fields. Now both are done in fix_session_vcol_expr().
>
> We can do cleanup at the end of every statement and fix_fields at the
> beginning of a statement. Just like for normal non-persistent items.
> For every vcol that needs it. This way they can safely use execution
> arena.

Nice catch! Done. Btw, expr_arena was already used in
update_virtual_fields(), update_default_fields(), set_default(). So
the leak is already there and now we just didn't make it bigger. I
marked leak places with TODO comments.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Aleksey Midenkov]

Hi Sergei!

On Thu, Jan 6, 2022 at 7:13 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Could you please explain in the commit comment what the bug was and what
> you are fixing?

I have added a new message to the last commit. This will be the
message of the squashed patch.

>
> I wasn't able to understad that, so I had to debug those crashes myself.
>
> So, the reason is, objects allocated in the wrong arena, right?
> And it was done inconsistently, on different code paths the "re-fix"
> context was different.

Yes.

>
> I agree that encapsulating all the context in one Vcol_expr_context is a
> good idea, it'll allow to have the same, correct, context everywhere.
>
> But always using expr_arena doesn't necessarily seem to be a correct
> choice. E.g. Arg_comparator creates Item_cache_temporal on every
> fix_field. If vcol expr is re-fixed all the time, expr_arena will grow
> continuously. Same for charset conversion, CASE, IN, etc. I suspect that
> if the vcol expr is always re-fixed, it has to use stmt arena.

stmt_arena cannot be used as long as 'expr' item is created on
expr_arena. 'expr' item is created at the vcol expression parse and is
reused during the whole TABLE lifetime. If some of the containee items
later were allocated on stmt_arena, 'expr' item will try to access
them in a different statement when they are already released. That is
what this bugfix tries to address.

Yes, this fix introduces a small and in most cases imperceptible
memory leak which can be easily overcame by FLUSH TABLES. The fix for
this leak is non-trivial.

1. vcol expr is not always refixed, but conditionally controlled by
vcols_need_refixing. And we must remove this control and truly refix
always.

2. we must clone 'expr' item into statement mem_root at each statement.

I have attached the leak fix PoC into this email. I do not know
whether this is worth it.

>
> Why do you re-fix fields? I can hardly imagine that an Item_field can
> find itself in a different table. And if it's always in the same table,
> why to force it to find this table over and over?

That was the alias fix. Refix of Item_field was required. I returned
back cleanup_excluding_fields_processor() and removed VCOL_TABLE_ALIAS
processing despite my opinion to apply that.

>
> We've already discussed VCOL_TABLE_ALIAS, no need to go there again.
>
> On Jan 06, Aleksey Midenkov wrote:
> > revision-id: 93493a0e9b5 (mariadb-10.2.40-194-g93493a0e9b5)
> > parent(s): a565e63c54c
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-12-29 00:46:31 +0300
> > message:
> >
> > MDEV-24176 Server crashes after insert in the table with virtual
> > column generated using date_format() and if()
> >
> > When table is reopened from cache vcol_info contains stale
> > expression. We refresh expression via TABLE::vcol_fix_exprs() but
> > first we must prepare a proper context (Vcol_expr_context) and do some
> > preparations which meet some requirements:
> >
> > 1. fields must not be fixed, we unfix them with cleanup() via
> > cleanup_processor.
> >
> > 2. Item_ident must have cached_table cleared. Again, this is stale
> > data which was already freed. cached_table interferes with
> > find_field_in_tables().
> >
> > We cannot clear cached_table directly in Item_ident::cleanup()
> > method. Lots of spaghetti logic depends on cached_table existence even
> > after cleanup() done.
> >
> > 3. If Item_ident has table_name non-NULL we trigger expr update. That
> > is done via Item_ident::check_vcol_func_processor() and
> > VCOL_TABLE_ALIAS flag.
> >
> > (4-7) The below conditions are prepared by Vcol_expr_context and used in
> > both fix_session_expr_for_read() and TABLE::vcol_fix_exprs():
> >
> > 4. Any fix_expr() must be done on expr_arena as there may be new items
> > created. It was a bug in fix_session_expr_for_read(). It was just not
> > reproduced because there were no second refix. Now refix is done for
> > more cases so it does reproduce. Tests affected: vcol.binlog
> >
> > 5. Also name resolution context must be narrowed to the single table
> > with all crutches in place. Tests affected: vcol.update
> >
> > 6. sql_mode must be clean and not fail expr update.
> >
> > sql_mode such as MODE_NO_BACKSLASH_ESCAPES, MODE_NO_ZERO_IN_DATE, etc
> > must not affect vcol expression update. If the table was created
> > successfully any further evaluation must not fail. Tests affected:
> > main.func_like
> >
> > 7. Warnings are disabled during expr update. It is assumed that these
> > warnings were printed before during initialization phase or later
> > during exe

Re: [Maria-developers] ef70e809a40: MDEV-27217 DELETE partition selection doesn't work for history partitions

[Aleksey Midenkov]

Hi Sergei!

On Wed, Jan 12, 2022 at 8:23 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jan 12, Aleksey Midenkov wrote:
> > revision-id: ef70e809a40 (mariadb-10.3.31-80-gef70e809a40)
> > parent(s): 88f8aa20bba
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-12-23 04:15:39 +0300
> > message:
> >
> > MDEV-27217 DELETE partition selection doesn't work for history partitions
> >
> > LIMIT history switching requires the number of history partitions to
> > be marked for read: from first to last non-empty plus one empty. The
> > least we can do is to fail with error message if the needed partition
> > was not marked for read. As this is handler interface we require new
> > handler error code to display user-friendly error message.
> >
> > Switching by INTERVAL works out-of-the-box with
> > ER_ROW_DOES_NOT_MATCH_GIVEN_PARTITION_SET error.
>
> Code-wise it's pretty much fine.
> Two problems related to error messages:
>
> We cannot have part of the error message a hard-coded English phrase.
> It doesn't work with localized error messages:
>
>"Невозможно выбрать раздел: modifying system versioned table"
>
> Also we cannot add new error messages in 10.3. Or any old GA.
> Currently you can add new error messages to 10.6 or any later version.
>
> You can reuse ER_UNUSED_1 for your new error message, that'll work.
> And, because you cannot have English phrases in it, I'd suggest
> something more generic, like
>
>   Not allowed for system-versioned table %`s.%`s
>
> In the context of
>
>  delete from t1 partition (p1);
>  ERROR HY000: Not allowed for system-versioned table `test`.`t1`
>
> it seems to be quite clear.

Fixed. Please have a glance again.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 8a84f5a40c1: MDEV-24176 Preparations

[Aleksey Midenkov]

Hi Sergei,

On Mon, Jan 3, 2022 at 2:44 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Dec 29, Aleksey Midenkov wrote:
> > > > > > +  /*
> > > > > > + NOTE: came from TABLE::alias_name_used and this is only a 
> > > > > > hint! It works
> > > > > > + only in need_correct_ident() condition. On other cases it is 
> > > > > > FALSE even if
> > > > > > + table_name is alias! It cannot be TRUE in these cases, lots 
> > > > > > of spaghetti
> > > > > > + logic depends on that.
> > > > >
> > > > > could you elaborate on that?
> > > I see that. What I mean was, can you show an example?
> > > Say, a query that sets TABLE::alias_name_used incorrectly?
> > > I wasn't able to find a test case for that.
> >
> > That's the original test case of the task. It has alias_name_used
> > false. Well, early I had more problems with that. Now I did the PoC
> > fix again and I see no big problems with the tests
> > (bb-10.2-midenok-tmp).
>
> Does it mean, you'll remove the comment (and workarounds, if any) in
> your branch?

Actually, no. That means it is possible to fix alias_name_used. I just
removed "It cannot be TRUE in these cases, lots of spaghetti logic
depends on that." But I now suspect if I go further in fixing, it will
turn out to be also true.


>
> > > Also, why do you even want to re-fix items when a table alias is
> > > used? Is it for MDEV-25672? But that's already fixed for half a
> > > year.
> >
> > Originally this fix was done before the pushed one. And do you really
> > want to keep bad value in Item_field? Even if you avoid using it now
> > someone surely will suffer from that after some new development or
> > when new use cases happen. If there is a ready fix that eliminates
> > such an accident in the future why don't you want to push it?
>
> I generally prefer not to do work that's not needed. That's called lazy :)
> If there's a value that's not used, ever, then I would suggest not to
> spend time cleaning it up. Re-preparing vcols after every DML that
> happened to use table aliases - that's not lazy, that's too much work.

Lazy update is the pattern to address specific performance issues. I
don't believe it should be used everywhere or at every convenient
occasion. I'm not really sure if Item_field was designed to keep dirty
values. That at least deserves a comment.

>
> As far as MDEV-25672 is concerned - making ALTER TABLE to check that
> existing vcols refer to the correct table - that's too much work too,
> we already know they do. Only new columns need to be checked.

Agree about new columns, but not about "much work". Performance
optimization should be done where it does matter. If something is done
per-row, that is always performance-sensitive. In the per-statement
case we may or may not optimize, that depends on specific cases and
trade-offs.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



--
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 8a84f5a40c1: MDEV-24176 Preparations

[Aleksey Midenkov]

Hi Sergei!

On Tue, Dec 28, 2021 at 12:03 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Oct 18, Aleksey Midenkov wrote:
> > On Sun, Oct 17, 2021 at 4:55 PM Sergei Golubchik  wrote:
> > > On Oct 17, Aleksey Midenkov wrote:
> > > > commit 8a84f5a40c1
> > > > Author: Aleksey Midenkov 
> > > > Date:   Thu May 27 17:00:14 2021 +0300
> > > >
> > > > MDEV-24176 Preparations
> > > >
> > > > 1. moved fix_vcol_exprs() call to open_table()
> > > >
> > > > mysql_alter_table() doesn't do lock_tables() so it cannot win from
> > > > fix_vcol_exprs() from there. Tests affected: main.default_session
> > >
> > > This is likely wrong, but the old code was wrong too. Neither open_table
> > > nor lock_tables is called under LOCK TABLES, but the session
> > > environment can change, I suspect.
> >
> > Why, open_table() is called under LOCK TABLES. Please see there `if
> > (best_table)` and a jump to reset where vcol_fix_exprs() is called.
> > Added test case for LOCK TABLES.
>
> Yes, you're right
>
> > > > 3. Vanilla cleanups and comments.
> > > >
> > > > diff --git a/sql/item.h b/sql/item.h
> > > > index cc1914a7ad4..7b7fe04f0b2 100644
> > > > --- a/sql/item.h
> > > > +++ b/sql/item.h
> > > > @@ -2586,6 +2585,12 @@ class Item_ident :public Item_result_field
> > > >const char *db_name;
> > > >const char *table_name;
> > > >const char *field_name;
> > > > +  /*
> > > > + NOTE: came from TABLE::alias_name_used and this is only a hint! 
> > > > It works
> > > > + only in need_correct_ident() condition. On other cases it is 
> > > > FALSE even if
> > > > + table_name is alias! It cannot be TRUE in these cases, lots of 
> > > > spaghetti
> > > > + logic depends on that.
> > >
> > > could you elaborate on that?
> >
> > See in next commit check_vcol_func_processor(). I could use `if
> > (alias_name_used)` instead of `if (table_name)` if alias_name_used
> > were updated correctly.
>
> I see that. What I mean was, can you show an example?
> Say, a query that sets TABLE::alias_name_used incorrectly?
> I wasn't able to find a test case for that.

That's the original test case of the task. It has alias_name_used
false. Well, early I had more problems with that. Now I did the PoC
fix again and I see no big problems with the tests
(bb-10.2-midenok-tmp).

>
> Also, why do you even want to re-fix items when a table alias is used?
> Is it for MDEV-25672? But that's already fixed for half a year.

Originally this fix was done before the pushed one. And do you really
want to keep bad value in Item_field? Even if you avoid using it now
someone surely will suffer from that after some new development or
when new use cases happen. If there is a ready fix that eliminates
such an accident in the future why don't you want to push it?

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



--
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b95c5105e28: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi, Sergei!

On Mon, Dec 6, 2021 at 11:22 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Dec 05, Aleksey Midenkov wrote:
> > > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc
> > > > index ac22a63bdba..459edb614f7 100644
> > > > --- a/sql/sql_base.cc
> > > > +++ b/sql/sql_base.cc
> > > > @@ -4203,6 +,15 @@ bool open_tables(THD *thd, const DDL_options_st 
> > > > ,
> > > >  }
> > > >
> > > >thd->current_tablenr= 0;
> > > > +
> > > > +#ifdef WITH_PARTITION_STORAGE_ENGINE
> > > > +  if (!thd->stmt_arena->is_conventional())
> > > > +  {
> > > > +for (tables= *start; tables; tables= tables->next_global)
> > > > +  tables->vers_skip_create= false;
> > > > +  }
> > >
> > > This looks a bit like an overkill, you do it for every prepare,
> > > while auto-adding a partition is a rare event, as you said.
> > >
> > > May be you can do instead in TABLE::vers_switch_partition
> > >
> > >  uint *create_count= !thd->stmt_arena->is_conventional() &&
> > >  table_list->vers_skip_create ?
> >
> > No, this is inside backoff action loop. That will interfere with the
> > normal vers_skip_create algorithm (which also applies to PS, SP). The
> > goal of the above code is to reset vers_skip_create from the previous
> > statement execution.
>
> In that case you can use an auto-resetting value, like,
> if tables->vers_skip_create == thd->query_id it means, "skip".
> That is vers_skip_create must be of query_id_t. You set it with
> tables->vers_skip_create= thd->query_id;
> And on the next statement it automatically expires.
>
> This semantics is a bit more complex than boolean, so it'd need a
> comment, like
>
>   /*
>   Protect single thread from repeating partition auto-create over
>   multiple share instances (as the share is closed on backoff action).
>   Skips auto-create only for one given query id.
>   */
>   query_id_tvers_skip_create;

Done! Thanks for the tip.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b95c5105e28: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei,

On Tue, Nov 23, 2021 at 11:29 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> This is a review of diff b95c5105e2 2552bce607
>
> This was quite good, thanks! Just a couple of questions/comments, see
> below.
>
> On Nov 23, Aleksey Midenkov wrote:
>
> > diff --git a/mysql-test/suite/versioning/t/partition.test 
> > b/mysql-test/suite/versioning/t/partition.test
> > index 006a65e1a16..a5bb8426ae4 100644
> > --- a/mysql-test/suite/versioning/t/partition.test
> > +++ b/mysql-test/suite/versioning/t/partition.test
> > @@ -1070,15 +1080,572 @@ alter table t1 add partition partitions 2;
> ...
> > +--let $datadir= `select @@datadir`
> > +--let $dummy= $datadir/test/t1#P#p1.ibd
> > +--write_file $dummy
> > +EOF
> > +
> > +set timestamp= unix_timestamp('2000-01-01 01:00:00');
> > +--error ER_GET_ERRNO
> > +update t1 set x= x + 2;
>
> That's a good one!
>
> > +show warnings;
> > +--remove_file $dummy
> > +
> > diff --git a/sql/table.h b/sql/table.h
> > index 2e074abcea0..d861db261bf 100644
> > --- a/sql/table.h
> > +++ b/sql/table.h
> > @@ -910,6 +911,11 @@ struct TABLE_SHARE
> >vers_kind_t versioned;
> >period_info_t vers;
> >period_info_t period;
> > +  /*
> > +  Protect multiple threads from repeating partition auto-create over
> > +  single share.
>
> could you please add here (to the comment)
>
>  TODO remove it when partitioning metadata will be in TABLE_SHARE

Added.

>
> > +  */
> > +  bool  vers_skip_auto_create;
> >
> >bool init_period_from_extra2(period_info_t *period, const uchar *data,
> > const uchar *end);
> > @@ -2557,6 +2567,11 @@ struct TABLE_LIST
> >bool  merged;
> >bool  merged_for_insert;
> >bool  sequence;  /* Part of NEXTVAL/CURVAL/LASTVAL */
> > +  /*
> > +  Protect single thread from repeating partition auto-create over
> > +  multiple share instances (as the share is closed on backoff action).
> > +  */
> > +  bool  vers_skip_create;
>
> I don't understand it. What does it mean "multiple share instances"?
> Can there be multiple TABLE_SHARE objects for the same table? How?

Over the time. The share is released and then reacquired.

>
> >
> >/*
> >  Items created by create_view_field and collected to change them in case
> > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> > index d7bb02bbd4e..56d5a68efd0 100644
> > --- a/sql/sql_partition.cc
> > +++ b/sql/sql_partition.cc
> > @@ -2587,11 +2587,15 @@ char *generate_partition_syntax(THD *thd, 
> > partition_info *part_info,
> >  err+= str.append(ctime, ctime_len);
> >  err+= str.append('\'');
> >}
> > +  if (vers_info->auto_hist)
> > +err+= str.append(STRING_WITH_LEN(" AUTO"));
> >  }
> > -if (vers_info->limit)
> > +else if (vers_info->limit)
> >  {
> >err+= str.append(STRING_WITH_LEN("LIMIT "));
> >err+= str.append_ulonglong(vers_info->limit);
> > +  if (vers_info->auto_hist)
> > +err+= str.append(STRING_WITH_LEN(" AUTO"));
>
> Can you do this str.appent("AUTO") once, after the if()?
> Not in both branches?

Done.


>
> >  }
> >}
> >else if (part_info->part_expr)
> > diff --git a/sql/partition_info.cc b/sql/partition_info.cc
> > index fd92e437cac..ddcc6f413ba 100644
> > --- a/sql/partition_info.cc
> > +++ b/sql/partition_info.cc
> > @@ -865,9 +870,167 @@ void partition_info::vers_set_hist_part(THD *thd)
> >  {
> >vers_info->hist_part= next;
> >if (next->range_value > thd->query_start())
> > -return;
> > +  {
> > +error= false;
> > +break;
> > +  }
> > +}
> > +if (error)
> > +{
> > +  if (auto_hist)
> > +  {
> > +*create_count= 0;
> > +const my_time_t hist_end= (my_time_t) 
> > vers_info->hist_part->range_value;
> > +DBUG_ASSERT(thd->query_start() >= hist_end);
> > +MYSQL_TIME h0, q0;
> > +my_tz_OFFSET0->gmt_sec_to_TIME(, hist_end);
> > +my_tz_OFFSET0->gmt_sec_to_TIME(, thd->query_start());
> > +longlong q= pack_time();
> > +longlong h= pack_time();
> > +while (h <= q)
> > +{
> > +  if (dat

Re: [Maria-developers] 8a84f5a40c1: MDEV-24176 Preparations

[Aleksey Midenkov]

Hi Sergei!

On Sun, Oct 17, 2021 at 4:55 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Oct 17, Aleksey Midenkov wrote:
> > commit 8a84f5a40c1
> > Author: Aleksey Midenkov 
> > Date:   Thu May 27 17:00:14 2021 +0300
> >
> > MDEV-24176 Preparations
> >
> > 1. moved fix_vcol_exprs() call to open_table()
> >
> > mysql_alter_table() doesn't do lock_tables() so it cannot win from
> > fix_vcol_exprs() from there. Tests affected: main.default_session
>
> This is likely wrong, but the old code was wrong too. Neither open_table
> nor lock_tables is called under LOCK TABLES, but the session
> environment can change, I suspect.

Why, open_table() is called under LOCK TABLES. Please see there `if
(best_table)` and a jump to reset where vcol_fix_exprs() is called.
Added test case for LOCK TABLES.

>
> >
> > 2. cleanup_excluding_fields_processor removed.
> >
> > That was just a quick hack to exclude wrongly working Item_field from
> > processing. Now it works due to correct execution environment (see
> > next commit). Related to MDEV-10355
>
> does it work now, in that commit, or only after the next commit?

It is for the next commit actually. Moved that there.


> what exactly do you mean by correct execution environment?

This is what Vcol_expr_context does.

>
> >
> > 3. Vanilla cleanups and comments.
> >
> > diff --git a/sql/item.h b/sql/item.h
> > index cc1914a7ad4..7b7fe04f0b2 100644
> > --- a/sql/item.h
> > +++ b/sql/item.h
> > @@ -2586,6 +2585,12 @@ class Item_ident :public Item_result_field
> >const char *db_name;
> >const char *table_name;
> >const char *field_name;
> > +  /*
> > + NOTE: came from TABLE::alias_name_used and this is only a hint! It 
> > works
> > + only in need_correct_ident() condition. On other cases it is FALSE 
> > even if
> > + table_name is alias! It cannot be TRUE in these cases, lots of 
> > spaghetti
> > + logic depends on that.
>
> could you elaborate on that?

See in next commit check_vcol_func_processor(). I could use `if
(alias_name_used)` instead of `if (table_name)` if alias_name_used
were updated correctly. But it is used only for limited subset of
cases (so is not always true when alias is specified). Improving
alias_name_used caused me some failures (in find_field_in_tables()
AFAIR), so I had to mark VCOL_TABLE_ALIAS for every existing
Item_ident::table_name. That triggers redundant expr update for some
cases.

>
> > +  */
> >bool alias_name_used; /* true if item was resolved against alias */
> >/*
> >  Cached value of index for this field in table->field array, used by 
> > prep.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



--
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] eb121bbe93e: Vanilla cleanups and refactorings

[Aleksey Midenkov]

Hi, Sergey!

On Sat, Sep 11, 2021 at 12:45 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 10, Aleksey Midenkov wrote:
> > On Fri, Sep 10, 2021 at 9:32 PM Sergei Golubchik  wrote:
> > > On Sep 10, Aleksey Midenkov wrote:
> > > > revision-id: eb121bbe93e (mariadb-10.6.1-78-geb121bbe93e)
> > > > parent(s): d90f54e2206
> > > > author: Aleksey Midenkov
> > > > committer: Aleksey Midenkov
> > > > timestamp: 2021-09-10 17:42:10 +0300
> > > > message:
> > > >
> > > > diff --git a/dbug/tests.c b/dbug/tests.c
> > > > index 3e77bf82236..100388891eb 100644
> > > > --- a/dbug/tests.c
> > > > +++ b/dbug/tests.c
> > > > @@ -13,6 +13,17 @@ char *push1=0;
> > > >  #include 
> > > >  #include 
> > > >
> > > > +
> > > > +#ifndef DBUG_OFF
> > > > +#define DBUG_EVALUATE(keyword,a1,a2) \
> > > > +(_db_keyword_(0,(keyword), 0) ? (a1) : (a2))
> > > > +#define DBUG_EVALUATE_IF(keyword,a1,a2) \
> > > > +(_db_keyword_(0,(keyword), 1) ? (a1) : (a2))
> > > > +#else
> > > > +#define DBUG_EVALUATE(keyword,a1,a2) (a2)
> > > > +#define DBUG_EVALUATE_IF(keyword,a1,a2) (a2)
> > > > +#endif
> > > > +
> > >
> > > could you fix the test properly, please?
> >
> > Sorry, I couldn't do that quickly. Do you know the recipe for how this is 
> > done?
>
> I thought it's just deleting everything that uses DBUG_EVALUATE*.
> Let me try...
>
> I've pushed the commit into bb-10.7-serg - just take it and remove
>
> +#undef DBUG_EVALUATE
> +#undef DBUG_EVALUATE_IF
> +#define DBUG_IF(X) _db_keyword_(0,X,1)
>
> from tests.c - that'll do.

No, that fails. Feel free to amend my branch. I'm going to be offline now.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 818407474d6: MDEV-22166 CONVERT PARTITION: move out partition into a table

[Aleksey Midenkov]

Btw, Sergei,

that MacOS problem in MDEV-22165 is not a showstopper, so I didn't
investigate that. The main business was to get the task into
usable/reviewable state. Probably that bug can be reproduced on
vanilla, so I change my answer to "I don't know".

On Sat, Sep 11, 2021 at 12:24 AM Aleksey Midenkov  wrote:
>
> Hi Sergei, Monty!
>
> On Fri, Sep 10, 2021 at 10:02 PM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey!
> >
> > I presume that changes to the logging code is what Monty asked.
>
> Not everything. I've done what I could in the current timeframe, the
> less important things were postponed. Didn't get his approval for that
> though, so I'm asking now whether the current state of patch is OK to
> push from his POV?
>
> > So I'll only comment on a couple of minor issues:
> >
> > On Sep 10, Aleksey Midenkov wrote:
> > > revision-id: 818407474d6 (mariadb-10.6.1-80-g818407474d6)
> > > parent(s): bb4b95975f7
> > > author: Aleksey Midenkov
> > > committer: Aleksey Midenkov
> > > timestamp: 2021-09-10 18:47:31 +0300
> > > message:
> > >
> > > MDEV-22166 CONVERT PARTITION: move out partition into a table
> > >
> > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> > > index db371d23ba7..cbdb40472d8 100644
> > > --- a/sql/sql_partition.cc
> > > +++ b/sql/sql_partition.cc
> > > @@ -5371,8 +5367,12 @@ that are reorganised.
> > >  tab_part_info->is_auto_partitioned= FALSE;
> > >}
> > >  }
> > > -else if (alter_info->partition_flags & ALTER_PARTITION_DROP)
> > > +else if ((alter_info->partition_flags & ALTER_PARTITION_DROP) |
> > > + (alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT))
> > >  {
> > > +  const char * const cmd=
> > > +(alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT) ?
> > > +  "EXTRACT" : "DROP";
> >
> > "CONVERT"
>
> Fixed.
>
> >
> > >/*
> > >  Drop a partition from a range partition and list partitioning is
> > >  always safe and can be made more or less immediate. It is 
> > > necessary
> > > diff --git a/sql/sql_table.h b/sql/sql_table.h
> > > index aacb6c99f15..a580c7d348d 100644
> > > --- a/sql/sql_table.h
> > > +++ b/sql/sql_table.h
> > > @@ -20,6 +20,17 @@
> > >  #include  // pthread_mutex_t
> > >  #include "m_string.h"   // LEX_CUSTRING
> > >
> > > +inline bool suicide()
> > > +{
> > > +  DBUG_SUICIDE();
> > > +  return false;
> > > +}
> >
> > as I wrote earlier, this is wrong. DBUG_SUICIDE() must always be an
> > expression.
>
> Fixed.
>
> >
> > > +
> > > +#define ERROR_INJECT_CRASH(code) \
> > > +  (DBUG_IF(code) && suicide())
> > > +#define ERROR_INJECT_ERROR(code) \
> > > +  (DBUG_IF(code) && (my_error(ER_UNKNOWN_ERROR, MYF(0)), 1))
> > > +
> > >  class Alter_info;
> > >  class Alter_table_ctx;
> > >  class Column_definition;
> >
> > Regards,
> > Sergei
> > VP of MariaDB Server Engineering
> > and secur...@mariadb.org
>
>
>
> --
> @midenok



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 818407474d6: MDEV-22166 CONVERT PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei, Monty!

On Fri, Sep 10, 2021 at 10:02 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> I presume that changes to the logging code is what Monty asked.

Not everything. I've done what I could in the current timeframe, the
less important things were postponed. Didn't get his approval for that
though, so I'm asking now whether the current state of patch is OK to
push from his POV?

> So I'll only comment on a couple of minor issues:
>
> On Sep 10, Aleksey Midenkov wrote:
> > revision-id: 818407474d6 (mariadb-10.6.1-80-g818407474d6)
> > parent(s): bb4b95975f7
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-09-10 18:47:31 +0300
> > message:
> >
> > MDEV-22166 CONVERT PARTITION: move out partition into a table
> >
> > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> > index db371d23ba7..cbdb40472d8 100644
> > --- a/sql/sql_partition.cc
> > +++ b/sql/sql_partition.cc
> > @@ -5371,8 +5367,12 @@ that are reorganised.
> >  tab_part_info->is_auto_partitioned= FALSE;
> >}
> >  }
> > -else if (alter_info->partition_flags & ALTER_PARTITION_DROP)
> > +else if ((alter_info->partition_flags & ALTER_PARTITION_DROP) |
> > + (alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT))
> >  {
> > +  const char * const cmd=
> > +(alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT) ?
> > +  "EXTRACT" : "DROP";
>
> "CONVERT"

Fixed.

>
> >/*
> >  Drop a partition from a range partition and list partitioning is
> >  always safe and can be made more or less immediate. It is necessary
> > diff --git a/sql/sql_table.h b/sql/sql_table.h
> > index aacb6c99f15..a580c7d348d 100644
> > --- a/sql/sql_table.h
> > +++ b/sql/sql_table.h
> > @@ -20,6 +20,17 @@
> >  #include  // pthread_mutex_t
> >  #include "m_string.h"   // LEX_CUSTRING
> >
> > +inline bool suicide()
> > +{
> > +  DBUG_SUICIDE();
> > +  return false;
> > +}
>
> as I wrote earlier, this is wrong. DBUG_SUICIDE() must always be an
> expression.

Fixed.

>
> > +
> > +#define ERROR_INJECT_CRASH(code) \
> > +  (DBUG_IF(code) && suicide())
> > +#define ERROR_INJECT_ERROR(code) \
> > +  (DBUG_IF(code) && (my_error(ER_UNKNOWN_ERROR, MYF(0)), 1))
> > +
> >  class Alter_info;
> >  class Alter_table_ctx;
> >  class Column_definition;
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] eb121bbe93e: Vanilla cleanups and refactorings

[Aleksey Midenkov]

Hi, Sergei!

On Fri, Sep 10, 2021 at 9:32 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 10, Aleksey Midenkov wrote:
> > revision-id: eb121bbe93e (mariadb-10.6.1-78-geb121bbe93e)
> > parent(s): d90f54e2206
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-09-10 17:42:10 +0300
> > message:
> >
> > diff --git a/dbug/tests.c b/dbug/tests.c
> > index 3e77bf82236..100388891eb 100644
> > --- a/dbug/tests.c
> > +++ b/dbug/tests.c
> > @@ -13,6 +13,17 @@ char *push1=0;
> >  #include 
> >  #include 
> >
> > +
> > +#ifndef DBUG_OFF
> > +#define DBUG_EVALUATE(keyword,a1,a2) \
> > +(_db_keyword_(0,(keyword), 0) ? (a1) : (a2))
> > +#define DBUG_EVALUATE_IF(keyword,a1,a2) \
> > +(_db_keyword_(0,(keyword), 1) ? (a1) : (a2))
> > +#else
> > +#define DBUG_EVALUATE(keyword,a1,a2) (a2)
> > +#define DBUG_EVALUATE_IF(keyword,a1,a2) (a2)
> > +#endif
> > +
>
> could you fix the test properly, please?
>

Sorry, I couldn't do that quickly. Do you know the recipe for how this is done?

> >  const char *func3()
> >  {
> >DBUG_ENTER("func3");
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei,

On Tue, Sep 7, 2021 at 12:34 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 06, Aleksey Midenkov wrote:
> > > >
> > > > I believe ALTER TABLE atomicity is not the perfect one in respect of
> > > > rollback on error so why should that be an example for me?
> > >
> > > let's start from a statement. You're stating that ALTER TABLE atomicity
> > > is not the perfect one in respect of rollback on error.
> > >
> > > What do you mean by that? Can you show how ALTER TABLE wouldn't be
> > > atomic after a rollback on an error?
> >
> > An example test is attached to this email.
>
> Hmm, I see, thanks.
>
> > I guess partitioning has no approach for that, the best it does is
> > printing the warning message. So it is 2 of them. And partitioning can
> > be easily switched to my scheme. As for the other DDL, it should be
> > simplified as well, I hope this is possible. But as an intermediate we
> > can have 2 approaches: for partitioning (my scheme) and for other DDL.
>
> Okay, can you push this MDEV into preview-10.7-MDEV-22166-convert-partition ?
> Only commits related to this MDEV, properly logically squashed, tests
> fixed. Commits related to CREATE OR REPLACE shouldn't be there.
> changes to parser.test or any traces of EXTRACT neither. I suspect you
> broke dbug-t unit test, please, verify that it passes.

Pushed.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Sergei,

On Wed, Sep 1, 2021 at 8:51 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Aug 27, Aleksey Midenkov wrote:
> > >
> > > I still don't quite understand why did you introduce a new method
> > > with frm backup and a (low probability) window where it won't clean
> > > up.
> > >
> > > Isn't it similar to normal ALTER TABLE? It also needs to replace
> > > frms and it is solved with ddl log - if a crash happens after the
> > > point of no return (after the frm was replaced with a new one), ddl
> > > log will binlog the query on recovery.
> > >
> > > Could you do the same, like
> > >
> > >   case DDL_LOCK_MIGRATE_PARTITION_ACTION:
> > > if (shadow frm no longer exists && xid not in binlog)
> > >   write_bin_log(...);
> > >
> > > would that work? Or would partitioning-specific old ddl logging code
> > > interfere?
> >
> > I believe ALTER TABLE atomicity is not the perfect one in respect of
> > rollback on error so why should that be an example for me?
>
> let's start from a statement. You're stating that ALTER TABLE atomicity
> is not the perfect one in respect of rollback on error.
>
> What do you mean by that? Can you show how ALTER TABLE wouldn't be
> atomic after a rollback on an error?
>

An example test is attached to this email.

> > Another issue with the scheme you proposed is worse complexity and
> > reliability.
>
> It's the scheme that ALTER TABLE and other DDL statements are using now.
> If you have something simpler and more reliable, could you please
> describe it so that we could change all DDL statements to use your
> approach?
>
> > Why do I have to do some vague logic about binlogging if it can be
> > straightforward and simple? So either I have to expand frm handling or
> > add another DDL log action: why are you asking me not to do the former
> > and do the latter?
>
> Because - as far as I understand - the latter is what all other DDL
> statements use, except old partitioning statements that weren't
> converted yet.
>
> we really do not want *three* different approaches to atomicity in the
> same ALTER TABLE statement. It's just not maintainable.

I guess partitioning has no approach for that, the best it does is
printing the warning message. So it is 2 of them. And partitioning can
be easily switched to my scheme. As for the other DDL, it should be
simplified as well, I hope this is possible. But as an intermediate we
can have 2 approaches: for partitioning (my scheme) and for other DDL.

>
> But perhaps I misunderstood and DDL atomicity was implemented
> differently from what I've described? Then I'm sorry, I didn't intend to
> suggest that you should implement yet another approach to DDL atomicity,
> I was saying, please, do what other statements do.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok


z.test
Description: Binary data
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei!

On Thu, Sep 2, 2021 at 5:13 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > On Wed, Sep 1, 2021 at 10:20 PM Sergei Golubchik  wrote:
> > > On Sep 01, Aleksey Midenkov wrote:
> > > > > >
> > > > > > Looks like IF_DBUG is superfluous macro and should be replaced by
> > > > > >
> > > > > > #ifndef DBUG_OFF
> > > > > > #endif
> > > > >
> > > > > No, it's used in expression. Precisely, to avoid ifdefs.
> > > >
> > > > So, what about DBUG(A) variant?
> > >
> > > We have IF_XXX for many different XXX. IF_DBUG was created to follow
> > > this convention. Anytime you see a macro IF_XXX you know what it does.
> > > Let's keep it that way.
> >
> > We also use DBUG_ prefix for debug things. IF_DBUG() scheme is
> > superfluous as only a1 used in most cases. So if renamed to DBUG_DO()
> > or just to DBUG() the scheme is not broken and the code looks nicer.
> > Please have a glance at the patch attached.
>
> I did. I also grepped include/ for IF_[A-Z]+, we have
>
> IF_WIN, IF_EMBEDDED, IF_PARTITIONING, IF_WSREP, IF_DBUG, IF_VALGRIND
>
> together they're used on 165 lines. It would be very unhelpful, if
> IF_DBUG would suddenly deviate from this scheme and would become a
> special exception to remember.

Okay, I'm not going to change that. Though IF_DBUG() AFAIR is used
naturally only in a couple of lines. The dozen of other IF_DBUG() uses
are only with one argument.

So, do you still think DBUG_IF() makes confusion with IF_DBUG()?

>
> > > > > So only one existing error message uses NOT_EXIST without DOES.
> > > > > Let's keep the conventional naming. So, it should be
> > > > >
> > > > >ER_KEY_DOES_NOT_EXIST
> > > > >ER_KEY_COLUMN_DOES_NOT_EXIST
> > > > >ER_PARTITION_DOES_NOT_EXIST
> > > > >ER_REORG_PARTITION_DOES_NOT_EXIST
> > > >
> > > > That is longer by the whole useless word...
> > >
> > > It correct grammar. Messages looking bad otherwise.
> >
> > But we are talking only about code names and they are not visible to a
> > user, are they?
> > Even SQL itself does use the short form "NOT EXISTS". :)
>
> it's IF NOT EXISTS, but it does indeed.
> Okay, if you keep backward compatibility defines, you
> can rename them.

Updated.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org

--
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei!

On Sat, Sep 4, 2021 at 3:34 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Aug 26, Sergei Golubchik wrote:
> > > >
> > > > ALTER ... MOVE TABLE table_name TO PARTITION (...)
> > > > ALTER ... MOVE PARTITION ... TO TABLE table_name
> > > >
> > > There is no MOVE symbol and we should not add a new one because of
> > > the same compatibility reason: it will not be possible to create
> > > table "move" (as well as to work with existing one I suspect).
> >
> > So, yes, I totally agree that it's best to avoid adding new keywords
> > to the grammar.
> >
> > > That's why we found MIGRATE as a good candidate And the reasons to
> > > have some special keyword, not just "ADD PARTITION ... FROM ..." are:
> > >
> > If it's my suggested synax from above, but with MIGRATE, it becomes
> >
> >   ALTER TABLE ... MIGRATE TABLE tbl_name TO PARTITION (partition definition)
> >   ALTER TABLE ... MIGRATE PARTITION part_name TO TABLE tbl_name
> >
> > so very symmetrical.
>
> "MIGRATE" is still a bit awkward here, because "migrate" is historically
> an intransitive verb, it isn't normally used like that.
>
> I went over all keywords in lex.h again and I think this one works
> quite well:
>
>ALTER TABLE ... CONVERT TABLE tbl_name TO PARTITION (partition definition)
>ALTER TABLE ... CONVERT PARTITION part_name TO TABLE tbl_name
>
> I also think it conveys the meaning better.

Yes, CONVERT looks more natural. Updated the code.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] dc2ace70f1b: Syntax with MIGRATE keyword

[Aleksey Midenkov]

Sergei,

On Wed, Sep 1, 2021 at 10:35 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > On Wed, Sep 1, 2021 at 10:13 PM Sergei Golubchik  wrote:
> > > On Sep 01, Aleksey Midenkov wrote:
> > > > > >
> > > > > > +move_out_partition:
> > > > > > +MIGRATE_SYM PARTITION_SYM
> > > > > > +| MIGRATE_SYM OUT_SYM PARTITION_SYM
> > > > >
> > > > > Where does this OUT_SYM came out from?
> > > > > It doesn't add any values, it doesn't make the syntax more natural.
> > > > > Why did you add it?
> > > >
> > > > I think that helps to make syntax more natural if we do MIGRATE
> > > > PARTITION in both directions.
> > > > If it is written MIGRATE OUT or MIGRATE IN that is easier to
> > > > understand what's going on. FROM/TO in the end is not helping much
> > > > because it is harder to notice. Partition specification presence or
> > > > absence: not so explicit to understand quickly.
> > >
> > > It incorrect English. And it's a noise word that adds no value and
> > > doesn't make anything easier to understand.
> > >
> > >   MIGRATE PARTITION x TO TABLE y
> > >
> > > is correct and unambigous although not a very traditional use of the
> > > word "migrate". MIGRATE OUT PARTITION x TO TABLE y is just wrong.
> >
> > If there is "move in" and "move out", "migrate in" and "migrate out"
> > cannot be wrong. Look for examples for "migrate out to" and "migrate
> > in from".
>
> Move would be a correct word here, as migrate is normally an
> intransitive word. Anyway, "out" is used similarly for them, it'd be
>
>MIGRATE PARTITION x OUT TO TABLE y
>

The above is not helping much. Ok, let's remove OUT.

> is that what you want? This looks kind of redundant, but I cannot say it
> is wrong. But MIGRATE OUT PARTITION x TO TABLE y - this is clearly
> broken.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Sergei,

On Wed, Sep 1, 2021 at 10:20 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > > >
> > > > Looks like IF_DBUG is superfluous macro and should be replaced by
> > > >
> > > > #ifndef DBUG_OFF
> > > > #endif
> > >
> > > No, it's used in expression. Precisely, to avoid ifdefs.
> >
> > So, what about DBUG(A) variant?
>
> We have IF_XXX for many different XXX. IF_DBUG was created to follow
> this convention. Anytime you see a macro IF_XXX you know what it does.
> Let's keep it that way.

We also use DBUG_ prefix for debug things. IF_DBUG() scheme is
superfluous as only a1 used in most cases. So if renamed to DBUG_DO()
or just to DBUG() the scheme is not broken and the code looks nicer.
Please have a glance at the patch attached.

>
> > >
> > > About your ER_KEY_COLUMN_DOES_NOT_EXITS replacement:
> > >
> > > $ grep -c DOES_NOT_EXIST sql/share/errmsg-utf8.txt
> > > 6
> > > $ grep -c NOT_EXIST sql/share/errmsg-utf8.txt
> > > 7
> > >
> > > So only one existing error message uses NOT_EXIST without DOES.
> > > Let's keep the conventional naming. So, it should be
> > >
> > >ER_KEY_DOES_NOT_EXIST
> > >ER_KEY_COLUMN_DOES_NOT_EXIST
> > >ER_PARTITION_DOES_NOT_EXIST
> > >ER_REORG_PARTITION_DOES_NOT_EXIST
> >
> > That is longer by the whole useless word...
>
> It correct grammar. Messages looking bad otherwise.

But we are talking only about code names and they are not visible to a
user, are they?
Even SQL itself does use the short form "NOT EXISTS". :)
I don't see any problems here.

>
> Or, better, don't rename error messages at all and
> preserve the compatibility with older applications.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok


u.diff.gz
Description: application/gzip
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] dc2ace70f1b: Syntax with MIGRATE keyword

[Aleksey Midenkov]

Sergei,

On Wed, Sep 1, 2021 at 10:13 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > > >
> > > > +move_out_partition:
> > > > +MIGRATE_SYM PARTITION_SYM
> > > > +| MIGRATE_SYM OUT_SYM PARTITION_SYM
> > >
> > > Where does this OUT_SYM came out from?
> > > It doesn't add any values, it doesn't make the syntax more natural.
> > > Why did you add it?
> >
> > I think that helps to make syntax more natural if we do MIGRATE
> > PARTITION in both directions.
> > If it is written MIGRATE OUT or MIGRATE IN that is easier to
> > understand what's going on. FROM/TO in the end is not helping much
> > because it is harder to notice. Partition specification presence or
> > absence: not so explicit to understand quickly.
>
> It incorrect English. And it's a noise word that adds no value and
> doesn't make anything easier to understand.
>
>   MIGRATE PARTITION x TO TABLE y
>
> is correct and unambigous although not a very traditional use of the
> word "migrate". MIGRATE OUT PARTITION x TO TABLE y is just wrong.
>

If there is "move in" and "move out", "migrate in" and "migrate out"
cannot be wrong. Look for examples for "migrate out to" and "migrate
in from".

> > > > +;
> > > > +
> > > > +to_table:
> > > > +TO_SYM TABLE_SYM
> > > > +| TO_SYM
> > >
> > > Let's keep the TABLE_SYM. Saying
> > >
> > >MIGRATE PARTITION p1 TO p2
> > >
> > > is very vague, the syntax should explicitly say that it migrates a
> > > PARTITION to a TABLE. The statement mentions many different types of
> > > objects, if should be clear about what object type every identifier
> > > refers to.
> >
> > That is a shortcut for easier typing and that is precious for
> > command-line work! If some names can make misunderstanding one can
> > always write
> >
> > MIGRATE PARTITION p1 TO TABLE p2
> >
> > Please do not restrict user freedom to choose between verbosity and
> > brevity. I guess you have a preference for perl over python (and the
> > democracy over dictatorship).
>
> I do. But it's SQL, not perl. SQL is very verbose even when it isn't
> really justified. And here is is justified, so let's keep the syntax
> unambigous.

And why this is good and should be followed? There are shortcuts in SQL too.
Okay, let's keep this unambiguous for a while.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Sergei,

On Wed, Sep 1, 2021 at 9:07 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > >
> > > We have a series of IF_xxx(A,B) macros, IF_PARTITIONING, IF_WIN, etc.
> > > IF_DBUG(A,B) expands to A if dbug is compiled in, otherwise into B.
> > >
> > > having IF_DBUG(A,B) and DBUG_IF(keyword) might be confusing.
> >
> > Looks like IF_DBUG is superfluous macro and should be replaced by
> >
> > #ifndef DBUG_OFF
> > #endif
>
> No, it's used in expression. Precisely, to avoid ifdefs.

So, what about DBUG(A) variant?

>
> > > > > > > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> > > > > > > > index 85d880cbbb4..4172a61812e 100644
> > > > > > > > --- a/sql/sql_partition.cc
> > > > > > > > +++ b/sql/sql_partition.cc
> > > >
> > > > Renamed this error code and ugly ER_KEY_COLUMN_DOES_NOT_EXITS
> > > > Changed the message.
> > >
> > > Unfortunatey ER_xxx constants are generally part of the API, client
> > > applications do
> > >
> > >if (mysql_errno() == ER_KEY_COLUMN_DOES_NOT_EXITS) ...
> > >
> > > so renaming cannot be done lightly. Only if really unavoidable and then
> > > with a compatibility fallback, like we have now in mysql.h:
> > >
> > >   #define ER_WARN_DATA_TRUNCATED WARN_DATA_TRUNCATED
> > >   #define WARN_PLUGIN_DELETE_BUILTIN ER_PLUGIN_DELETE_BUILTIN
> > >   #define ER_FK_DUP_NAME ER_DUP_CONSTRAINT_NAME
> > >   ...
> >
> > I'd make that anyway with that proxy stuff in mysql.h.
> > C++14 can mark variables deprecated. That can help throw away
> > deprecated constants after some period of time.
>
> Those are defines, I don't know if you can mark them deprecated.
>
> About your ER_KEY_COLUMN_DOES_NOT_EXITS replacement:
>
> $ grep -c DOES_NOT_EXIST sql/share/errmsg-utf8.txt
> 6
> $ grep -c NOT_EXIST sql/share/errmsg-utf8.txt
> 7
>
> So only one existing error message uses NOT_EXIST without DOES.
> Let's keep the conventional naming. So, it should be
>
>ER_KEY_DOES_NOT_EXIST
>ER_KEY_COLUMN_DOES_NOT_EXIST
>ER_PARTITION_DOES_NOT_EXIST
>ER_REORG_PARTITION_DOES_NOT_EXIST

That is longer by the whole useless word...

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] dc2ace70f1b: Syntax with MIGRATE keyword

[Aleksey Midenkov]

Sergei,

On Wed, Sep 1, 2021 at 8:21 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > revision-id: dc2ace70f1b (mariadb-10.6.1-68-gdc2ace70f1b)
> > parent(s): 19fbfab084f
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-08-30 22:37:08 +0300
> > message:
> >
> > Syntax with MIGRATE keyword
> >
> > ALTER TABLE tbl_name
> > [alter_option [, alter_option] ...] |
> > [partition_options]
> >
> > partition_option: {
> > ...
> > | MIGRATE [OUT] PARTITION partition_name TO [TABLE] tbl_name
> > }
> >
> > Examples:
> >
> > ALTER TABLE t1 MIGRATE PARTITION p1 TO tp1;
> > ALTER TABLE t1 MIGRATE PARTITION p2 TO TABLE tp2;
> > ALTER TABLE t1 MIGRATE OUT PARTITION p3 TO TABLE tp3;
> >
> > diff --git a/mysql-test/suite/parts/r/alter_table.result 
> > b/mysql-test/suite/parts/r/alter_table.result
> > index bc1bf661d4c..7a2d45c51c1 100644
> > --- a/mysql-test/suite/parts/r/alter_table.result
> > +++ b/mysql-test/suite/parts/r/alter_table.result
> > @@ -258,9 +258,46 @@ show grants for current_user;
> >  Grants for alan@%
> >  GRANT USAGE ON *.* TO `alan`@`%`
> >  GRANT INSERT, CREATE, DROP, ALTER ON `test`.* TO `alan`@`%`
> > -alter table t1 extract partition p1 as table tp1;
> > +alter table t1 migrate partition p1 to table tp1;
> >  disconnect alan;
> >  connection default;
> >  drop database EXISTENT;
> >  drop user alan;
> >  drop tables t1, tp1, tp2, tp4;
> > +# Cunning syntax
>
> what's so cunning here?
>
> > +create or replace table t1 (x int)
> > +partition by range(x) (
> > diff --git a/sql/handler.h b/sql/handler.h
> > index c455625aafe..7ac9e929ecc 100644
> > --- a/sql/handler.h
> > +++ b/sql/handler.h
> > @@ -824,7 +824,7 @@ typedef bool Log_func(THD*, TABLE*, bool, const uchar*, 
> > const uchar*);
> >  #define ALTER_PARTITION_TRUNCATE(1ULL << 11)
> >  // Set for REORGANIZE PARTITION
> >  #define ALTER_PARTITION_TABLE_REORG   (1ULL << 12)
> > -#define ALTER_PARTITION_EXTRACT(1ULL << 13)
> > +#define ALTER_PARTITION_MIGRATE_OUT(1ULL << 13)
> >
> >  /*
> >This is master database for most of system tables. However there
> > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy
> > index 49dd244f2ff..e1be6ea9557 100644
> > --- a/sql/sql_yacc.yy
> > +++ b/sql/sql_yacc.yy
> > @@ -7492,6 +7492,16 @@ ident_or_empty:
> >  | ident
> >  ;
> >
> > +move_out_partition:
> > +MIGRATE_SYM PARTITION_SYM
> > +| MIGRATE_SYM OUT_SYM PARTITION_SYM
>
> Where does this OUT_SYM came out from?
> It doesn't add any values, it doesn't make the syntax more natural.
> Why did you add it?

I think that helps to make syntax more natural if we do MIGRATE
PARTITION in both directions.
If it is written MIGRATE OUT or MIGRATE IN that is easier to
understand what's going on. FROM/TO in the end is not helping much
because it is harder to notice. Partition specification presence or
absence: not so explicit to understand quickly.

>
> > +;
> > +
> > +to_table:
> > +TO_SYM TABLE_SYM
> > +| TO_SYM
>
> Let's keep the TABLE_SYM. Saying
>
>MIGRATE PARTITION p1 TO p2
>
> is very vague, the syntax should explicitly say that it migrates a
> PARTITION to a TABLE. The statement mentions many different types of
> objects, if should be clear about what object type every identifier
> refers to.

That is a shortcut for easier typing and that is precious for
command-line work! If some names can make misunderstanding one can
always write

MIGRATE PARTITION p1 TO TABLE p2

Please do not restrict user freedom to choose between verbosity and
brevity. I guess you have a preference for perl over python (and the
democracy over dictatorship).

>
> > +;
> > +
> >  alter_commands:
> >/* empty */
> >  | DISCARD TABLESPACE
> > @@ -7619,15 +7629,15 @@ alter_commands:
> >MYSQL_YYABORT;
> >  Lex->alter_info.partition_flags|= ALTER_PARTITION_EXCHANGE;
> >}
> > -| EXTRACT_SYM PARTITION_SYM alt_part_name_item
> > -  AS TABLE_SYM table_ident have_partitioning
> > +| move_out_partition alt_part_name_item
> > +  to_table table_ident have_partitioning
>
> please, let it be just
>
>  MIGRATE_SYM PARTITION_SYM alt_part_name_item
>  TO_SYM TABLE_SYM table_ident have_partitioning

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi, Sergei!

On Wed, Sep 1, 2021 at 2:58 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Aug 30, Aleksey Midenkov wrote:
> > Hi Sergei!
> >
> > Updated bb-10.7-midenok-MDEV-22166 is d4668e7254c6
>
> I'll look right after replying to this email.
>
> > > Okay, indeed, it seems that DBUG_EVALUATE_IF is almost always used with
> > > one of the arguments being 0 or 1. If you replace all
> > > DBUG_EVALUATE_IF's, let's just remove it completely.
> > > Note, that DBUG_TRUE_IF can be defined simply as
> > >
> > >   #define DBUG_TRUE_IF(keyword)  _db_keyword_(0, (keyword), 1)
> > >
> > > so may be you'd like to call id DBUG_KEYWORD_IF or something?
> > > Although DBUG_TRUE_IF is shorter :)
> >
> > It turned out DBUG_IF() is even more short! ;)
>
> We have a series of IF_xxx(A,B) macros, IF_PARTITIONING, IF_WIN, etc.
> IF_DBUG(A,B) expands to A if dbug is compiled in, otherwise into B.
>
> having IF_DBUG(A,B) and DBUG_IF(keyword) might be confusing.
>

Looks like IF_DBUG is superfluous macro and should be replaced by

#ifndef DBUG_OFF
#endif

But for some code like here

IF_DBUG(uchar const *const old_pack_ptr= pack_ptr;,)

I'd make

DBUG(uchar const *const old_pack_ptr= pack_ptr);

So DBUG(A) macro would be a one-liner for #ifndef DBUG_OFF.

> > > > > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> > > > > > index 85d880cbbb4..4172a61812e 100644
> > > > > > --- a/sql/sql_partition.cc
> > > > > > +++ b/sql/sql_partition.cc
> >
> > Renamed this error code and ugly ER_KEY_COLUMN_DOES_NOT_EXITS
> > Changed the message.
>
> Unfortunatey ER_xxx constants are generally part of the API, client
> applications do
>
>if (mysql_errno() == ER_KEY_COLUMN_DOES_NOT_EXITS) ...
>
> so renaming cannot be done lightly. Only if really unavoidable and then
> with a compatibility fallback, like we have now in mysql.h:
>
>   #define ER_WARN_DATA_TRUNCATED WARN_DATA_TRUNCATED
>   #define WARN_PLUGIN_DELETE_BUILTIN ER_PLUGIN_DELETE_BUILTIN
>   #define ER_FK_DUP_NAME ER_DUP_CONSTRAINT_NAME
>   ...
>

I'd make that anyway with that proxy stuff in mysql.h.
C++14 can mark variables deprecated. That can help throw away
deprecated constants after some period of time.

> > > > > >  goto err;
> > > > > >}
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 19fbfab084f: Compilation fix

[Aleksey Midenkov]

Hi Sergei!

On Wed, Sep 1, 2021 at 3:46 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Sep 01, Aleksey Midenkov wrote:
> > revision-id: 19fbfab084f (mariadb-10.6.1-67-g19fbfab084f)
> > parent(s): 8009680fbc3
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-08-30 22:02:40 +0300
> > message:
> >
> > Compilation fix
> >
> > diff --git a/sql/sql_table.h b/sql/sql_table.h
> > index 83e88c82f0e..8cb9ebe0445 100644
> > --- a/sql/sql_table.h
> > +++ b/sql/sql_table.h
> > @@ -20,8 +20,14 @@
> >  #include  // pthread_mutex_t
> >  #include "m_string.h"   // LEX_CUSTRING
> >
> > +inline bool suicide()
> > +{
> > +  DBUG_SUICIDE();
> > +  return false;
> > +}
> > +
> >  #define ERROR_INJECT_CRASH(code) \
> > -  (DBUG_IF(code) && (DBUG_SUICIDE(), 0))
> > +  (DBUG_IF(code) && suicide())
>
> What did that change? A compiler didn't like (void,0) ?

Here

#define DBUG_SUICIDE()  do { } while(0)

DBUG_SUICIDE is not an expression. It's a statement and cannot be used
in expression.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei!

Updated bb-10.7-midenok-MDEV-22166 is d4668e7254c6

On Thu, Aug 26, 2021 at 9:58 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Some replies.
>
> On Aug 26, Aleksey Midenkov wrote:
> > > > diff --git a/sql/lex.h b/sql/lex.h
> > > > index cbf9d9d51b2..55eefc39a7e 100644
> > > > --- a/sql/lex.h
> > > > +++ b/sql/lex.h
> > > > @@ -240,6 +240,7 @@ SYMBOL symbols[] = {
> > > >{ "EXPLAIN", SYM(DESCRIBE)},
> > > >{ "EXTENDED",SYM(EXTENDED_SYM)},
> > > >{ "EXTENT_SIZE", SYM(EXTENT_SIZE_SYM)},
> > > > +  { "EXTRACT", SYM(EXTRACT_SYM)},
> > >
> > > So, you did EXTRACT after all, not MIGRATE?
> >
> > I did like the original description said but I agree that this should be
> > changed to something better. Moreover EXTRACT parsing is now
> > incorrect, see
> >
> > git show 3b693cd3d9 mysql-test/main/parser.result
>
> Yes, I've noticed, didn't comment because we still discuss the syntax.
> If we'll decide to use EXTRACT after all, then perhaps it could be made
> non-reserved.
>
> > > ADD/EXTRACT aren't opposites in SQL.
> > >
> > > It would be rather consistent to have
> > >
> > > ALTER ... ADD PARTITION (...) FROM table_name
> > >
> > > and
> > >
> > > ALTER ... DROP PARTITION ... TO table_name
> > >
> > > except that DROP...TO looks kind of strange, and ADD...FROM
> > > does not convey the meaning that the table disappears.
> > >
> > > The second issue also applies to IMPORT/EXPORT.
> > >
> > > Another option, also kind of strange, but also consistent and
> > > makes it clear that the table/partition disappears:
> > >
> > > ALTER ... MOVE TABLE table_name TO PARTITION (...)
> > > ALTER ... MOVE PARTITION ... TO TABLE table_name
> > >
> > > or the same with CHANGE instead of MOVE to reuse an existing keyword.
> >
> > There is no MOVE symbol and we should not add a new one because of the
> > same compatibility reason: it will not be possible to create table
> > "move" (as well as to work with existing one I suspect).
>
> it will be, if MOVE won't be a reserved work.
> But still, keywords, even non-reserved, make the parser a bit slower.
> And may be a keyword in that context has to be reserved?
>
> So, yes, I totally agree that it's best to avoid adding new keywords to
> the grammar.
>
> > That's why we found MIGRATE as a good candidate And the reasons to
> > have some special keyword, not just "ADD PARTITION ... FROM ..." are:
> >
> > 1. it is not clear whether the source table stays or not (that was the
> > discussion in MDEV-22165);
> > 2. for keyword like MIGRATE it is clear how it works in both
> > directions and no need to remember or guess the opposite keyword;
> > 3. (and most important) is scalability: we may want to expand the
> > syntax for copying the table instead of moving it so instead of
> > MIGRATE we put COPY and that's it: same formula, clear how it works,
> > easy to interchange between MIGRATE and COPY.
> >
> > I've already thought about MOVE as a better keyword but sadly enough
> > it is not reserved. So how about the proposed syntax with MIGRATE?
>
> MIGRATE is not reserved, by the way, it's a non-reserved keyword.
>
> If it's my suggested synax from above, but with MIGRATE, it becomes
>
>   ALTER TABLE ... MIGRATE TABLE tbl_name TO PARTITION (partition definition)
>   ALTER TABLE ... MIGRATE PARTITION part_name TO TABLE tbl_name
>
> so very symmetrical.
>
>   ALTER TABLE ...  ADD PARTITION ... MIGRATE FROM TABLE ...
>
> looks kind of backward to me. At least if the reverse is MIGRATE PARTITION TO 
> TABLE.
>
> May be MIGRATE PARTITION TO TABLE and MIGRATE PARTITION FROM TABLE ?
> Instead of MIGRATE TABLE FROM PARTITION.

Looks nice. Made this kind of syntax.

>
> > > I'm not saying anything of the above is perfect, and better variants are
> > > very much welcome. But pretty much anything is better than ADD/EXTRACT.
> > >
> > > >{ "FALSE",   SYM(FALSE_SYM)},
> > > >{ "FAST",SYM(FAST_SYM)},
> > > >{ "FAULTS",  SYM(FAULTS_SYM)},
> > > > diff --git a/include/my_dbug.h b/include/my_dbug.h
> > > > index e25bfcf28a7..cf969ce5750 100644
> > > > --- a/include/my_dbug.h
> > > > +++ b/include/my_dbug.h
> > > > @@ -104,

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei!

On Fri, Aug 27, 2021 at 3:23 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Aug 26, Aleksey Midenkov wrote:
> > Hi Sergei!
> >
> > The updated branch is cabd0530 bb-10.7-midenok-MDEV-22166
>
> Commits
>
>   cabd0530426 Parser refactoring
>   db69d4eb025 Cleanups
>
> are fine, thanks. The main change in the simplified diff is:
>
> > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> > --- a/sql/log_event_server.cc
> > +++ b/sql/log_event_server.cc
> > @@ -1299,7 +1299,7 @@ bool Query_log_event::write()
> >if (thd && thd->binlog_xid)
> >{
> >  *start++= Q_XID;
> > -int8store(start, thd->query_id);
> > +int8store(start, thd->binlog_xid);
>
> I agree that it looks correct. How did you find it?
> Does your patch introduce a case where thd->binlog_xid != thd->query_id ?

I just tried to update xid via query_id and failed. The interface of
updating xid is quite weird by having to copy query_id to binlog_xid
and then clear binlog_xid back. I don't understand why this is needed.

>
> >  start+= 8;
> >}
> >
> > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> > --- a/sql/sql_partition.cc
> > +++ b/sql/sql_partition.cc
> > @@ -7439,18 +7439,28 @@ uint fast_alter_partition_table(THD *thd, TABLE 
> > *table,
> >  alter_partition_extract(lpt) ||
> >  (frm_install= TRUE, FALSE) ||
> >  mysql_write_frm(lpt, WFRM_INSTALL_SHADOW|WFRM_BACKUP_ORIGINAL) ||
> >  log_partition_alter_to_ddl_log(lpt) ||
> >  (frm_install= FALSE, FALSE) ||
> > +((!thd->lex->no_write_to_binlog) &&
> > +  (thd->binlog_xid= thd->query_id,
> > +  ddl_log_update_xid(lpt->part_info, thd->binlog_xid),
> > +  write_bin_log(thd, false, thd->query(), thd->query_length()),
> > +  thd->binlog_xid= 0)) ||
>
> do I understand it correctly - if the crash happens before the binlog
> write, ddl log on recovery will undo everything. If the crash happens
> after, it'll see the xid in binlog and will not undo the operation?

Yes. Don't forget about error handling. We rollback everything on
error, that is important for atomicity too. And that was not done for
the main code of mysql_alter_table() AFAIK.

>
> > +/*
> > +   Note: this crash-point does not cleanup backup (see 
> > WFRM_DROP_BACKUP),
> > +   but this crash-point is low probability as ddl_log_complete() is
> > +   fast operation.
> > +*/
>
> I still don't quite understand why did you introduce a new method
> with frm backup and a (low probability) window where it won't clean up.
>
> Isn't it similar to normal ALTER TABLE? It also needs to replace frms
> and it is solved with ddl log - if a crash happens after the point of no
> return (after the frm was replaced with a new one), ddl log will binlog
> the query on recovery.
>
> Could you do the same, like
>
>   case DDL_LOCK_MIGRATE_PARTITION_ACTION:
> if (shadow frm no longer exists && xid not in binlog)
>   write_bin_log(...);
>
> would that work? Or would partitioning-specific old ddl logging code
> interfere?

I believe ALTER TABLE atomicity is not the perfect one in respect of
rollback on error so why should that be an example for me? Another
issue with the scheme you proposed is worse complexity and
reliability. Why do I have to do some vague logic about binlogging if
it can be straightforward and simple? So either I have to expand frm
handling or add another DDL log action: why are you asking me not to
do the former and do the latter? I can create the second chain for
cleaning up the leftover backups and that would be anyway better than
binlogging in 2 places.

>
> >  (ddl_log_complete(lpt->part_info), false) ||
> > -((!thd->lex->no_write_to_binlog) &&
> > - (write_bin_log(thd, FALSE,
> > -thd->query(), thd->query_length()), FALSE)) ||
> >  mysql_write_frm(lpt, WFRM_DROP_BACKUP) ||
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table

[Aleksey Midenkov]

Hi Sergei!

The updated branch is cabd0530 bb-10.7-midenok-MDEV-22166

On Wed, Aug 25, 2021 at 9:33 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> It's great that you made it atomic. Frankly, I didn't think of it, MDEV
> defines the feature as being exactly equivalent to the 3-command
> sequence, so non-atomic. But, of course, making it atomic makes perfect
> sense.
>
> See other comments below.
>
> On Aug 23, Aleksey Midenkov wrote:
> > commit 6768e3a2830
> > Author: Aleksey Midenkov 
> > Date:   Thu Jul 29 22:54:12 2021 +0300
> >
> > diff --git a/sql/lex.h b/sql/lex.h
> > index cbf9d9d51b2..55eefc39a7e 100644
> > --- a/sql/lex.h
> > +++ b/sql/lex.h
> > @@ -240,6 +240,7 @@ SYMBOL symbols[] = {
> >{ "EXPLAIN", SYM(DESCRIBE)},
> >{ "EXTENDED",SYM(EXTENDED_SYM)},
> >{ "EXTENT_SIZE", SYM(EXTENT_SIZE_SYM)},
> > +  { "EXTRACT", SYM(EXTRACT_SYM)},
>
> So, you did EXTRACT after all, not MIGRATE?

I did like the original description said but I agree that this should be
changed to something better. Moreover EXTRACT parsing is now
incorrect, see

git show 3b693cd3d9 mysql-test/main/parser.result

> ADD/EXTRACT aren't opposites in SQL.
>
> It would be rather consistent to have
>
> ALTER ... ADD PARTITION (...) FROM table_name
>
> and
>
> ALTER ... DROP PARTITION ... TO table_name
>
> except that DROP...TO looks kind of strange, and ADD...FROM
> does not convey the meaning that the table disappears.
>
> The second issue also applies to IMPORT/EXPORT.
>
> Another option, also kind of strange, but also consistent and
> makes it clear that the table/partition disappears:
>
> ALTER ... MOVE TABLE table_name TO PARTITION (...)
> ALTER ... MOVE PARTITION ... TO TABLE table_name
>
> or the same with CHANGE instead of MOVE to reuse an existing keyword.

There is no MOVE symbol and we should not add a new one because of the
same compatibility reason: it will not be possible to create table
"move" (as well as to work with existing one I suspect).

That's why we found MIGRATE as a good candidate And the reasons to
have some special keyword, not just "ADD PARTITION ... FROM ..." are:

1. it is not clear whether the source table stays or not (that was the
discussion in MDEV-22165);
2. for keyword like MIGRATE it is clear how it works in both
directions and no need to remember or guess the opposite keyword;
3. (and most important) is scalability: we may want to expand the
syntax for copying the table instead of moving it so instead of
MIGRATE we put COPY and that's it: same formula, clear how it works,
easy to interchange between MIGRATE and COPY.

I've already thought about MOVE as a better keyword but sadly enough
it is not reserved. So how about the proposed syntax with MIGRATE?

>
> I'm not saying anything of the above is perfect, and better variants are
> very much welcome. But pretty much anything is better than ADD/EXTRACT.
>
> >{ "FALSE",   SYM(FALSE_SYM)},
> >{ "FAST",SYM(FAST_SYM)},
> >{ "FAULTS",  SYM(FAULTS_SYM)},
> > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy
> > index dc2eb4425f0..a41ea7d3e4a 100644
> > --- a/sql/sql_yacc.yy
> > +++ b/sql/sql_yacc.yy
> > @@ -7605,6 +7605,25 @@ alter_commands:
> >  if (Lex->stmt_alter_table_exchange_partition($6))
> >MYSQL_YYABORT;
> >}
> > +| EXTRACT_SYM PARTITION_SYM alt_part_name_item
> > +  AS TABLE_SYM table_ident have_partitioning
> > +  {
> > +LEX *lex= Lex;
> > +lex->first_select_lex()->db= $6->db;
> > +if (lex->first_select_lex()->db.str == NULL &&
> > +lex->copy_db_to(>first_select_lex()->db))
> > +  MYSQL_YYABORT;
> > +if (unlikely(check_table_name($6->table.str,$6->table.length,
> > +  FALSE)) ||
> > +($6->db.str && unlikely(check_db_name((LEX_STRING*) 
> > &$6->db
> > +  my_yyabort_error((ER_WRONG_TABLE_NAME, MYF(0), 
> > $6->table.str));
>
> good point, LEX::stmt_alter_table_exchange_partition() doesn't seem to do it
>
> > +lex->name= $6->table;
> > +
> > +lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_table();
> > +if (unlikely(lex->m_sql_cmd == NULL))
> > +  MYSQL_YYABORT;
> > +lex->alter_info.partition_flags|= ALTER_PARTITION_EXTRACT;
>
> why woul

Re: [Maria-developers] f14cfd22115: Vanilla refactoring: binlog query in DDL_LOG_RENAME_ACTION

[Aleksey Midenkov]

Hi Sergei!

Really, this is not pure refactoring. I've been planning to remove it
if it will not be useful for MDEV-22165 (that task should be based on
this branch). Meanwhile, can you please review other commits?


On Mon, Aug 23, 2021 at 7:35 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Aug 23, Aleksey Midenkov wrote:
> > revision-id: f14cfd22115 (mariadb-10.6.1-63-gf14cfd22115)
> > parent(s): 03225474d60
> > author: Aleksey Midenkov
> > committer: Aleksey Midenkov
> > timestamp: 2021-08-17 20:01:48 +0300
> > message:
> >
> > Vanilla refactoring: binlog query in DDL_LOG_RENAME_ACTION
> >
> > This turned out to be not needed for MDEV-22166. But it still might be
> > useful for MDEV-22165 or later.
>
> There are three changes here.
> Adding 'locked' to ddl_log_store_query, creating
> st_ddl_recovery::binlog_query(), adding 'case RENAME_PHASE_BINLOG'
>
> st_ddl_recovery::binlog_query() is ok, if you want to keep it.
> But the other two aren't a refactoring, it's creating new functionality
> which isn't used in this patch, dead code basically. And your commit
> comment seems to imply that this new functionality isn't needed
> in the following feature patch either.
>
> While your other refactoring patches are indeed pure refactoring, this
> one isn't, so, please, remove it from the patchset if it isn't used
> there. You can keep st_ddl_recovery::binlog_query() if you'd like - it
> is indeed pure refactoring.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c34533186ab: MDEV-18734 ASAN heap-use-after-free upon sorting by blob column from partitioned table

[Aleksey Midenkov]

Hi Sergei!

On Mon, Jul 26, 2021 at 7:35 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jul 23, Aleksey Midenkov wrote:
> > Now updated the patch to
> >
> > git diff dfa2d0bc13..9b564832e3
>
> Thanks, no new comments about it. Just one question below still
>
> > > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
> > > > index c53732a2b51..70ed14448ea 100644
> > > > --- a/sql/ha_partition.cc
> > > > +++ b/sql/ha_partition.cc
> > > > @@ -5130,7 +5135,18 @@ bool ha_partition::init_record_priority_queue()
> > > >  i= bitmap_get_next_set(_part_info->read_partitions, i))
> > > >{
> > > >  DBUG_PRINT("info", ("init rec-buf for part %u", i));
> > > > -int2store(ptr, i);
> > > > +if (table->s->blob_fields)
> > > > +{
> > > > +  for (uint j= 0; j < table->s->blob_fields; ++j, ++objs)
> > > > +  {
> > > > +blob_storage[j]= new (objs) Ordered_blob_storage;
> > > > +if (!blob_storage[j])
> > > > +  DBUG_RETURN(true);
> > >
> > > is this possible?
> >
> > If we don't use exceptions we should handle OOM this way. If we can't
> > use -fno-exceptions we should make a wrapper by overloading default
> > new() and catching std::badalloc.
>
> I mean, it's a placement new, it doesn't allocate anything.
> How can it return NULL if it simply returns objs?

Oh, right! Updated that.

>
> ...
> > > > +else
> > > > +{
> > > > +  bool set_read_value;
> > > > +  String *cached= blob->cached(set_read_value);
> > > > +  if (cached)
> > > > +  {
> > > > +cached->swap(s.blob);
> > > > +s.set_read_value= set_read_value;
> > > > +  }
> > > > +}
> > > > +  }
> > > > +  table->move_fields(table->field, table->record[0], rec_buf);
> > >
> > > On swap_blobs() you always move all fields from rec_buf to 
> > > table->record[0]?
> > > Why?
> > > This looks very suspicious.
> >
> > Have you seen the first move_fields() call? We work on rec_buf and if
> > it is the same as record[0] move_fields() does nothing.
>
> Oops, no. I missed the first one. And it was very suspicious that you
> always move in one direction.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c34533186ab: MDEV-18734 ASAN heap-use-after-free upon sorting by blob column from partitioned table

[Aleksey Midenkov]

Hi Sergei!

Now updated the patch to

git diff dfa2d0bc13..9b564832e3

On Wed, Jul 21, 2021 at 11:33 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> As before, despite the email subject, it's
>
>   git diff dfa2d0bc13..e533b5fb307
>
> Don't be confused :)
>
> On Jul 21, Aleksey Midenkov wrote:
>
> > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
> > index c53732a2b51..70ed14448ea 100644
> > --- a/sql/ha_partition.cc
> > +++ b/sql/ha_partition.cc
> > @@ -5130,7 +5135,18 @@ bool ha_partition::init_record_priority_queue()
> >  i= bitmap_get_next_set(_part_info->read_partitions, i))
> >{
> >  DBUG_PRINT("info", ("init rec-buf for part %u", i));
> > -int2store(ptr, i);
> > +if (table->s->blob_fields)
> > +{
> > +  for (uint j= 0; j < table->s->blob_fields; ++j, ++objs)
> > +  {
> > +blob_storage[j]= new (objs) Ordered_blob_storage;
> > +if (!blob_storage[j])
> > +  DBUG_RETURN(true);
>
> is this possible?
>

If we don't use exceptions we should handle OOM this way. If we can't
use -fno-exceptions we should make a wrapper by overloading default
new() and catching std::badalloc.

> > +  }
> > +  *((Ordered_blob_storage ***) ptr)= blob_storage;
> > +  blob_storage+= table->s->blob_fields;
> > +}
> > +int2store(ptr + sizeof(String **), i);
> >  ptr+= m_priority_queue_rec_len;
> >}
> >m_start_key.key= (const uchar*)ptr;
> > @@ -6291,6 +6333,43 @@ int 
> > ha_partition::handle_ordered_index_scan_key_not_found()
> >  }
> >
> >
> > +void ha_partition::swap_blobs(uchar * rec_buf, Ordered_blob_storage ** 
> > storage, bool restore)
> > +{
> > +  uint *ptr, *end;
> > +  uint blob_n= 0;
> > +  table->move_fields(table->field, rec_buf, table->record[0]);
> > +  for (ptr= table->s->blob_field, end= ptr + table->s->blob_fields;
> > +   ptr != end; ++ptr, ++blob_n)
> > +  {
> > +DBUG_ASSERT(*ptr < table->s->fields);
> > +Field_blob *blob= (Field_blob*) table->field[*ptr];
> > +DBUG_ASSERT(blob->flags & BLOB_FLAG);
> > +DBUG_ASSERT(blob->field_index == *ptr);
> > +if (!bitmap_is_set(table->read_set, *ptr) || blob->is_null())
> > +  continue;
> > +
> > +Ordered_blob_storage = *storage[blob_n];
> > +
> > +if (restore)
> > +{
> > +  if (!s.blob.is_empty())
> > +blob->swap(s.blob, s.set_read_value);
>
> don't you need here something like
>
>   else
> blob->reset();
>
> to make blob empty?

I believe no. We protect only blob cache (value or read_value). If the
cache was empty that doesn't mean the blob was empty. AFAIR blobs
allocated by a storage engine should work just fine. Added a comment
about that.

>
> > +}
> > +else
> > +{
> > +  bool set_read_value;
> > +  String *cached= blob->cached(set_read_value);
> > +  if (cached)
> > +  {
> > +cached->swap(s.blob);
> > +s.set_read_value= set_read_value;
> > +  }
> > +}
> > +  }
> > +  table->move_fields(table->field, table->record[0], rec_buf);
>
> On swap_blobs() you always move all fields from rec_buf to table->record[0]?
> Why?
> This looks very suspicious.
>

Have you seen the first move_fields() call? We work on rec_buf and if
it is the same as record[0] move_fields() does nothing.

> > +}
> > +
> > +
> >  /*
> >Common routine to handle index_next with ordered results
> >
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei!

The rebased patches are in

bb-10.7-midenok-MDEV-17554

Do we have open questions on review?

On Thu, Jun 3, 2021 at 1:52 PM Aleksey Midenkov  wrote:
>
> Sergei,
>
> On Thu, Jun 3, 2021 at 12:27 PM Aleksey Midenkov  wrote:
> >
> > Sergei,
> >
> > On Thu, Jun 3, 2021 at 11:30 AM Sergei Golubchik  wrote:
> > >
> > > Hi, Aleksey!
> > >
> > > On Jun 03, Aleksey Midenkov wrote:
> > > > On Wed, Jun 2, 2021 at 11:37 PM Sergei Golubchik  
> > > > wrote:
> > > > > On Jun 02, Aleksey Midenkov wrote:
> > > > > > On Wed, Jun 2, 2021 at 9:06 PM Sergei Golubchik  
> > > > > > wrote:
> > > > > > >
> > > > > > > On Jun 02, Aleksey Midenkov wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -  if (!(sql_lock= (MYSQL_LOCK*)
> > > > > > > > > > > > > > - my_malloc(key_memory_MYSQL_LOCK, 
> > > > > > > > > > > > > > sizeof(*sql_lock) +
> > > > > > > > > > > > > > -   
> > > > > > > > > > > > > > sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2)
> > > > > > > > > > > > > >  +
> > > > > > > > > > > > > > -   
> > > > > > > > > > > > > > sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME
> > > > > > > > > > > > > > -DBUG_RETURN(0);  // 
> > > > > > > > > > > > > > Fatal error
> > > > > > > > > > > > > > +  const size_t lock_size= sizeof(*sql_lock) +
> > > > > > > > > > > > > > +sizeof(THR_LOCK_DATA *) * ((a->lock_count + 
> > > > > > > > > > > > > > b->lock_count) * 2) +
> > > > > > > > > > > > > > +sizeof(TABLE *) * (a->table_count + 
> > > > > > > > > > > > > > b->table_count);
> > > > > > > > > > > > > > +  if (thd)
> > > > > > > > > > > > > > +  {
> > > > > > > > > > > > > > +sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size);
> > > > > > > > > > > > > > +if (!sql_lock)
> > > > > > > > > > > > > > +  DBUG_RETURN(0);
> > > > > > > > > > > > > > +sql_lock->flags= GET_LOCK_ON_THD;
> > > > > > > > > > > > >
> > > > > > > > > > Yes, that is the part of the bug fix.
> > > > This was not a bug, this was a new change in recover_from_failed_open().
> > >
> > > Hmm, so was it part of the bug fix or not?
> >
> > The bug was it doesn't work with LTM_PRELOCKED. It was a part of the
> > bug fix. There were no bug with lock merging -- it was a new change to
> > satisfy that bug fix.
>
> If you are reviewing MDEV-23639, that part is not yet finished and
> fully pushed. Maybe there was a misunderstanding. Please wait until I
> reassign that ticket to you. I'll do that along with MDEV-17554 review
> changes.

That part was finished and the ticket is on you.

>
> >
> > >
> > > > > > Also freeing was impossible for locks on thd.
> > > > >
> > > > > Yes, this change I understand, no questions about freeing.
> > > > >
> > > > > > > > > > > > > > +  /*
> > > > > > > > > > > > > > +  NOTE: The semantics of vers_set_hist_part() 
> > > > > > > > > > > > > > is double:
> > > > > > > > > > > > >
> > > > > > > > > > > > > twofold
> > > > >
> > > > > Please, fix the language to be proper English.
> > > >
> > > > Don't you want to ask Ian now? Please look for "double semantics"
> > > > collocation in Google query (quotes are important). There are quite a
> > > > number of examples including scientific books and IETF drafts.
> > >
> > > No, I don't.
> > > "double semantics&qu

Re: [Maria-developers] b3844107287: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi Sergei!

On Tue, Jun 8, 2021 at 3:04 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 08, Aleksey Midenkov wrote:
> > revision-id: b3844107287 (mariadb-10.5.2-424-gb3844107287)
> > parent(s): 27d66d644cf
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2021-03-01 17:43:34 +0300
> > message:
> >
> > MDEV-16546 System versioning setting to allow history modification
> >
> > 1. force_fields_visible session variable makes system-invisible and
> > user-invisible fields visible on next table open. FLUSH TABLES
> > required for already opened tables.
>
> * system_versioning_insert_history
> * allows pseudocolumns ROW_START and ROW_END be specified in INSERT
> * doesn't affect invisible columns

Done. Probably the name "system_versioning_insert_history" may be
confused with something else. How about
"system_versioning_direct_history" or
"system_versioning_direct_insert"? Even
"system_versioning_modify_history" looks less confusing (and has space
for future extension).

> * FLUSH TABLES requirememnt is rather confusing, but let's have it,
>   at least until I could suggest something better

That was the requirement for force_fields_visible. For
system_versioning_insert_history there is no such requirement (as we
do not modify anything in TABLE_SHARE).

>
> > 2. If secure_timestamp allows to modify timestamp variable then
> > following DML commands: INSERT, INSERT..SELECT and LOAD DATA can
> > specify row_start and row_end system field values.
>
> when system_versioning_insert_history=1

Edited.

>
> > 3. Cleaned up select_insert::send_data() from setting vers_write as
> > this parameter is now set on TABLE initialization.
>
> I don't see where it's set. This commit removes two
>
>   table->vers_write= table->versioned();
>
> and adds one
>
>   from_field->table->vers_write= false;
>
> in Item_field::fix_fields. But where is vers_write set to true?
> Was it done in some other commit?

That was the commit 00a254cc for MDEV-20186 which sets vers_write in
TABLE::init().

>
> > diff --git a/mysql-test/suite/versioning/r/insert.result 
> > b/mysql-test/suite/versioning/r/insert.result
> > index 2645d0184e8..00d1bb705c4 100644
> > --- a/mysql-test/suite/versioning/r/insert.result
> > +++ b/mysql-test/suite/versioning/r/insert.result
> > @@ -112,3 +112,95 @@ xy
> >  99001
> >  drop table t1;
> >  drop table t2;
> > +#
> > +# MDEV-16546 System versioning setting to allow history modification
> > +#
> > +# restart: --secure-timestamp=NO
> > +set @@session.time_zone='+00:00';
> > +create table t1(x int) with system versioning;
> > +insert into t1(x) values (1);
> > +insert into t1(x, row_start, row_end) values (2, '1980-01-01 00:00:00', 
> > '1980-01-01 00:00:01');
> > +ERROR 42S22: Unknown column 'row_start' in 'field list'
> > +set session force_fields_visible= 1;
> > +flush tables;
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL,
> > +  `row_start` timestamp(6) GENERATED ALWAYS AS ROW START,
> > +  `row_end` timestamp(6) GENERATED ALWAYS AS ROW END,
>
> shouldn't be visible here, only in insert and load.

I removed force_fields_visible and hence this effect.

>
> > +  PERIOD FOR SYSTEM_TIME (`row_start`, `row_end`)
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > +insert into t1(x, row_start, row_end) values (3, '1980-01-01 00:00:00', 
> > '1980-01-01 00:00:01');
>
> please, add tests for
>
>insert into t1 values (4)
>
> this should work, row_start/row_end must be mentioned explicitly.

Added.

> Also
>
>insert into t1 set x=5, row_start=..., row_end=...
>
> this should work too.

Added.

> And when only one row_start or row_end is mentioned as well.
> But
>
>   update t1 set row_start=...
>   insert into t1 (x, row_start, row_end) values (...)
>  on duplicate key update row_start=...
>
> this should fail.

Ok.

> While
>
>   replace
>   load data ... replace
>
> should work, but as DELETE+INSERT (I mean that DELETE in versioned
> tables doesn't actually delete history).

Sorry, I don't understand the bracket part (why is it here). Normal
REPLACE is DELETE+INSERT, versioned REPLACE is UPDATE+INSERT. For
history replace only "normal REPLACE" is possible, so it is
DELETE+INSERT obviously.

Added REPLACE, but modifying row_end leads to a new row insert. See
this comment:

--echo # Changing row_end via REPLACE is NOT possible, we just insert new row

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()

[Aleksey Midenkov]

Hi Sergei!

On Thu, Jun 3, 2021 at 1:58 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jan 26, Aleksey Midenkov wrote:
> > On Tue, Dec 22, 2020 at 7:58 PM Sergei Golubchik  wrote:
> > > On Nov 03, Aleksey Midenkov wrote:
> > >
> > > And please, pretty please, don't use non-constant references as
> > > function arguments.
> >
> > Why? With pointers we can pass NULL and we don't need NULL there, do we?
>
> Because this is the policy. If you convince Monty that non-constant
> references are fine - feel free to use them.

If this is the policy, why there is

void String::swap(String )

since 2004? Are you asking me to pass the pointer and then dereference
it and pass it as a reference into String class? That would look weird
to me. I'm going to agree if you insist, but really?

>
> > > > On Mon, Nov 2, 2020 at 11:33 PM Sergei Golubchik  
> > > > wrote:
> > > > >
> > > > > > diff --git a/sql/field.cc b/sql/field.cc
> > > > > > index bdaaecc20269..0fd40c979d2c 100644
> > > > > > --- a/sql/field.cc
> > > > > > +++ b/sql/field.cc
> > > > > > @@ -8310,6 +8310,7 @@ int Field_blob::store(const char *from,uint 
> > > > > > length,CHARSET_INFO *cs)
> > > > > >copy_length= copier.well_formed_copy(field_charset,
> > > > > > (char*) value.ptr(), 
> > > > > > new_length,
> > > > > > cs, from, length);
> > > > > > +  value.length(copy_length);
> > > > >
> > > > > good! Could this be a distinct bug with its own test case?
> > > >
> > > > Do we need to spend time on that bureaucracy? There are real problems to
> > > > solve... Tickets drain time from the management people and generally 
> > > > from
> > > > anyone who sees them.
> > >
> > > I didn't mean opening an MDEV. Only writing a test case and extracting
> > > it into a separate commit.
> > >
> > > If this is a genuine bug fix and you don't create a test case - how can
> > > you be sure the fix will stay? It may disappear in a merge or a rebase,
> > > manually or automatically. Or in a refactoring that "looks fine, all
> > > tests pass". If you can create a test case - please, do.
> >
> > It is tested by federated.federated_partition test case.
>
> Do you mean, that if I remove this line, federated.federated_partition
> test will fail? If yes - good, this is the test I meant, no need to
> create another one.

Yes, the test will fail.

>
> > Why do we have to extract this side-fix to a separate commit?
>
> Because later when someone looks at the code it helps to be able to
> understand what a commit is actually fixing.

As long as both fixes are important and there is no sense to apply one
without the other I see separate commits as a measure to satisfy
someone's curiosity which I do not share (and there is nothing to be
curious about that one-liner). Though I'm going to agree if you ask me
to do that.

>
> > > > > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
> > > > > > --- a/sql/ha_partition.cc
> > > > > > +++ b/sql/ha_partition.cc
> > > > > > @@ -5125,14 +5128,14 @@ bool 
> > > > > > ha_partition::init_record_priority_queue()
> > > > > >uint alloc_len;
> > > > > >uint used_parts= bitmap_bits_set(_part_info->read_partitions);
> > > > > >/* Allocate record buffer for each used partition. */
> > > > > > -  m_priority_queue_rec_len= m_rec_length + PARTITION_BYTES_IN_POS;
> > > > > > +  m_priority_queue_rec_len= m_rec_length + ORDERED_REC_OFFSET;
> > > > > >if (!m_using_extended_keys)
> > > > > >m_priority_queue_rec_len += m_file[0]->ref_length;
> > > > > >alloc_len= used_parts * m_priority_queue_rec_len;
> > > > > >/* Allocate a key for temporary use when setting up the scan. */
> > > > > >alloc_len+= table_share->max_key_length;
> > > > > >
> > > > > > -  if (!(m_ordered_rec_buffer= (uchar*)my_malloc(alloc_len, 
> > > > > > MYF(MY_WME
> > > > > > +  if (!(m_ordered_rec_buffer= (uchar*) alloc_root(_ordered_root, 
> > > > > > alloc_len)))
> > > > >
> > > > > I don't see why you need a memroot here. memroo

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()

[Aleksey Midenkov]

Hi Sergei!

Do you plan to fix everything you just said and push yourself or let
me push it as is? Btw, there were many allocations over one loop:
that's why I used mem_root. multi_malloc() won't do the job there.
That could be done either with explicit moving pointer or some utility
class holding such pointer. I decided to upgrade mem_root for such
pattern so it could do the double job and we had a cleaner code.

On Thu, Jun 3, 2021 at 2:00 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> I think it's better to wrap it up quickly, agree on changes and push the
> fix before the next release.
>
> On Jun 01, Aleksey Midenkov wrote:
> > Hi Sergei!
> >
> > Users could benefit from this fix of MDEV-18734 since the last
> > changes. Do you really think the open questions left are so important
> > for this bug to be still not pushed?
> >
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Thu, Jun 3, 2021 at 12:27 PM Aleksey Midenkov  wrote:
>
> Sergei,
>
> On Thu, Jun 3, 2021 at 11:30 AM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey!
> >
> > On Jun 03, Aleksey Midenkov wrote:
> > > On Wed, Jun 2, 2021 at 11:37 PM Sergei Golubchik  wrote:
> > > > On Jun 02, Aleksey Midenkov wrote:
> > > > > On Wed, Jun 2, 2021 at 9:06 PM Sergei Golubchik  
> > > > > wrote:
> > > > > >
> > > > > > On Jun 02, Aleksey Midenkov wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > -  if (!(sql_lock= (MYSQL_LOCK*)
> > > > > > > > > > > > > - my_malloc(key_memory_MYSQL_LOCK, 
> > > > > > > > > > > > > sizeof(*sql_lock) +
> > > > > > > > > > > > > -   
> > > > > > > > > > > > > sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2)
> > > > > > > > > > > > >  +
> > > > > > > > > > > > > -   
> > > > > > > > > > > > > sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME
> > > > > > > > > > > > > -DBUG_RETURN(0);  // 
> > > > > > > > > > > > > Fatal error
> > > > > > > > > > > > > +  const size_t lock_size= sizeof(*sql_lock) +
> > > > > > > > > > > > > +sizeof(THR_LOCK_DATA *) * ((a->lock_count + 
> > > > > > > > > > > > > b->lock_count) * 2) +
> > > > > > > > > > > > > +sizeof(TABLE *) * (a->table_count + 
> > > > > > > > > > > > > b->table_count);
> > > > > > > > > > > > > +  if (thd)
> > > > > > > > > > > > > +  {
> > > > > > > > > > > > > +sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size);
> > > > > > > > > > > > > +if (!sql_lock)
> > > > > > > > > > > > > +  DBUG_RETURN(0);
> > > > > > > > > > > > > +sql_lock->flags= GET_LOCK_ON_THD;
> > > > > > > > > > > >
> > > > > > > > > Yes, that is the part of the bug fix.
> > > This was not a bug, this was a new change in recover_from_failed_open().
> >
> > Hmm, so was it part of the bug fix or not?
>
> The bug was it doesn't work with LTM_PRELOCKED. It was a part of the
> bug fix. There were no bug with lock merging -- it was a new change to
> satisfy that bug fix.

If you are reviewing MDEV-23639, that part is not yet finished and
fully pushed. Maybe there was a misunderstanding. Please wait until I
reassign that ticket to you. I'll do that along with MDEV-17554 review
changes.

>
> >
> > > > > Also freeing was impossible for locks on thd.
> > > >
> > > > Yes, this change I understand, no questions about freeing.
> > > >
> > > > > > > > > > > > > +  /*
> > > > > > > > > > > > > +  NOTE: The semantics of vers_set_hist_part() is 
> > > > > > > > > > > > > double:
> > > > > > > > > > > >
> > > > > > > > > > > > twofold
> > > >
> > > > Please, fix the language to be proper English.
> > >
> > > Don't you want to ask Ian now? Please look for "double semantics"
> > > collocation in Google query (quotes are important). There are quite a
> > > number of examples including scientific books and IETF drafts.
> >
> > No, I don't.
> > "double semantics" looks good, if you want to change the comment to
> >
> >   Note the double semantics of vers_set_hist_part() ...
> >
> > this is fine.
>
> And what's wrong with "is" construct?
> Note the red apple.
> Note the apple is red.
>
> The second one emphasizes the apple IS red. Is "double" some special
> adjective that cannot be used in the second form?
>
> >
> > > > > > > > > > > > > +  table_list->vers_skip_create= false;
> > > > > > > > > > > > >

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Thu, Jun 3, 2021 at 11:30 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 03, Aleksey Midenkov wrote:
> > On Wed, Jun 2, 2021 at 11:37 PM Sergei Golubchik  wrote:
> > > On Jun 02, Aleksey Midenkov wrote:
> > > > On Wed, Jun 2, 2021 at 9:06 PM Sergei Golubchik  
> > > > wrote:
> > > > >
> > > > > On Jun 02, Aleksey Midenkov wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > -  if (!(sql_lock= (MYSQL_LOCK*)
> > > > > > > > > > > > - my_malloc(key_memory_MYSQL_LOCK, 
> > > > > > > > > > > > sizeof(*sql_lock) +
> > > > > > > > > > > > -   
> > > > > > > > > > > > sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2)
> > > > > > > > > > > >  +
> > > > > > > > > > > > -   
> > > > > > > > > > > > sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME
> > > > > > > > > > > > -DBUG_RETURN(0);  // Fatal 
> > > > > > > > > > > > error
> > > > > > > > > > > > +  const size_t lock_size= sizeof(*sql_lock) +
> > > > > > > > > > > > +sizeof(THR_LOCK_DATA *) * ((a->lock_count + 
> > > > > > > > > > > > b->lock_count) * 2) +
> > > > > > > > > > > > +sizeof(TABLE *) * (a->table_count + 
> > > > > > > > > > > > b->table_count);
> > > > > > > > > > > > +  if (thd)
> > > > > > > > > > > > +  {
> > > > > > > > > > > > +sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size);
> > > > > > > > > > > > +if (!sql_lock)
> > > > > > > > > > > > +  DBUG_RETURN(0);
> > > > > > > > > > > > +sql_lock->flags= GET_LOCK_ON_THD;
> > > > > > > > > > >
> > > > > > > > Yes, that is the part of the bug fix.
> > This was not a bug, this was a new change in recover_from_failed_open().
>
> Hmm, so was it part of the bug fix or not?

The bug was it doesn't work with LTM_PRELOCKED. It was a part of the
bug fix. There were no bug with lock merging -- it was a new change to
satisfy that bug fix.

>
> > > > Also freeing was impossible for locks on thd.
> > >
> > > Yes, this change I understand, no questions about freeing.
> > >
> > > > > > > > > > > > +  /*
> > > > > > > > > > > > +  NOTE: The semantics of vers_set_hist_part() is 
> > > > > > > > > > > > double:
> > > > > > > > > > >
> > > > > > > > > > > twofold
> > >
> > > Please, fix the language to be proper English.
> >
> > Don't you want to ask Ian now? Please look for "double semantics"
> > collocation in Google query (quotes are important). There are quite a
> > number of examples including scientific books and IETF drafts.
>
> No, I don't.
> "double semantics" looks good, if you want to change the comment to
>
>   Note the double semantics of vers_set_hist_part() ...
>
> this is fine.

And what's wrong with "is" construct?
Note the red apple.
Note the apple is red.

The second one emphasizes the apple IS red. Is "double" some special
adjective that cannot be used in the second form?

>
> > > > > > > > > > > > +  table_list->vers_skip_create= false;
> > > > > > > > > > > > +  ot_ctx->vers_create_count= 0;
> > > > > > > > > > > > +  action= Open_table_context::OT_REOPEN_TABLES;
> > > > > > > > > > > > +  table_arg= NULL;
> > > > > > > > > > > > +}
> > > > > > > > > > >
> > > > > > > > > > > I'm afraid I don't understand. All this business with
> > > > > > > > > > > vers_skip_create and vers_skip_auto_create, it wasn't in
> > > > > > > > > > > the previous version of the patc

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Wed, Jun 2, 2021 at 11:37 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 02, Aleksey Midenkov wrote:
> > On Wed, Jun 2, 2021 at 9:06 PM Sergei Golubchik  wrote:
> > >
> > > On Jun 02, Aleksey Midenkov wrote:
> > > > > > > > > >
> > > > > > > > > > -  if (!(sql_lock= (MYSQL_LOCK*)
> > > > > > > > > > - my_malloc(key_memory_MYSQL_LOCK, sizeof(*sql_lock) +
> > > > > > > > > > -   
> > > > > > > > > > sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2) +
> > > > > > > > > > -   
> > > > > > > > > > sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME
> > > > > > > > > > -DBUG_RETURN(0);  // Fatal error
> > > > > > > > > > +  const size_t lock_size= sizeof(*sql_lock) +
> > > > > > > > > > +sizeof(THR_LOCK_DATA *) * ((a->lock_count + 
> > > > > > > > > > b->lock_count) * 2) +
> > > > > > > > > > +sizeof(TABLE *) * (a->table_count + b->table_count);
> > > > > > > > > > +  if (thd)
> > > > > > > > > > +  {
> > > > > > > > > > +sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size);
> > > > > > > > > > +if (!sql_lock)
> > > > > > > > > > +  DBUG_RETURN(0);
> > > > > > > > > > +sql_lock->flags= GET_LOCK_ON_THD;
> > > > > > > > >
> > > > > > Sorry, I meant the moment when MDEV-23639 calls mysql_lock_merge().
> > > > > > Yes, that is the part of the bug fix.
> > > > >
> > > > > What did it fix and how?
> > > > >
> > > > That's not about changing the method, that's about merging locks. When
> > > > I merged my locks with thd there were already thd-allocated locks.
> > >
> > > And? How did allocating locks on THD fix anything?
> >
> > LTM_PRELOCKED had locks on thd, I keep merged locks on thd. Isn't that
> > enough for you?
>
> I'm just trying to understand what the bug was. And I still cannot.

This was not a bug, this was a new change in recover_from_failed_open().

>
> > Also freeing was impossible for locks on thd.
>
> Yes, this change I understand, no questions about freeing.
>
> > > > > > > > > > +  /*
> > > > > > > > > > +  NOTE: The semantics of vers_set_hist_part() is 
> > > > > > > > > > double:
> > > > > > > > >
> > > > > > > > > twofold
>
> Please, fix the language to be proper English.

Don't you want to ask Ian now? Please look for "double semantics"
collocation in Google query (quotes are important). There are quite a
number of examples including scientific books and IETF drafts.

>
> > > > > > > > > > +  table_list->vers_skip_create= false;
> > > > > > > > > > +  ot_ctx->vers_create_count= 0;
> > > > > > > > > > +  action= Open_table_context::OT_REOPEN_TABLES;
> > > > > > > > > > +  table_arg= NULL;
> > > > > > > > > > +}
> > > > > > > > >
> > > > > > > > > I'm afraid I don't understand. All this business with
> > > > > > > > > vers_skip_create and vers_skip_auto_create, it wasn't in
> > > > > > > > > the previous version of the patch. So, I believe it was
> > > > > > > > > a fix for one of the MDEV bugs reported and fixed
> > > > > > > > > meanwhile.
> > > > > > > >
> > > > > > > > No, that was the multi-threaded case which worked good for
> > > > > > > > me, but suddenly I discovered it fails on some buildbot.
> > > > > > >
> > > > > > > Could you elaborate on what the problem was? Two threads
> > > > > > > trying to add the partition in parallel? I'd expect
> > > > > > > MDL_EXCLUSIVE to prevent that.
> > > > > >
> > > > > > MDL_EXCLUSIVE prevents parallel execution of
> > >

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Wed, Jun 2, 2021 at 9:06 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 02, Aleksey Midenkov wrote:
> > > > > > > >
> > > > > > > > -  if (!(sql_lock= (MYSQL_LOCK*)
> > > > > > > > - my_malloc(key_memory_MYSQL_LOCK, sizeof(*sql_lock) +
> > > > > > > > -   
> > > > > > > > sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2) +
> > > > > > > > -   
> > > > > > > > sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME
> > > > > > > > -DBUG_RETURN(0);  // Fatal error
> > > > > > > > +  const size_t lock_size= sizeof(*sql_lock) +
> > > > > > > > +sizeof(THR_LOCK_DATA *) * ((a->lock_count + b->lock_count) 
> > > > > > > > * 2) +
> > > > > > > > +sizeof(TABLE *) * (a->table_count + b->table_count);
> > > > > > > > +  if (thd)
> > > > > > > > +  {
> > > > > > > > +sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size);
> > > > > > > > +if (!sql_lock)
> > > > > > > > +  DBUG_RETURN(0);
> > > > > > > > +sql_lock->flags= GET_LOCK_ON_THD;
> > > > > > >
> > > > Sorry, I meant the moment when MDEV-23639 calls mysql_lock_merge().
> > > > Yes, that is the part of the bug fix.
> > >
> > > What did it fix and how?
> > >
> > That's not about changing the method, that's about merging locks. When
> > I merged my locks with thd there were already thd-allocated locks.
>
> And? How did allocating locks on THD fix anything?

LTM_PRELOCKED had locks on thd, I keep merged locks on thd. Isn't that
enough for you? Also freeing was impossible for locks on thd.

>
> > > > > > > > +  /*
> > > > > > > > +  NOTE: The semantics of vers_set_hist_part() is double:
> > > > > > >
> > > > > > > twofold
> > > > > >
> > > > Do you really believe in that butter butterish? :) I mean do we
> > > > need to discuss all sorts of butter? That level of white noise
> > > > should be ignored.
> > >
> > > I'm just trying to avoid incorrect usage of a language.
> >
> > I wish that threshold of incorrectness would be more relaxed.
>
> Sure. If any authoritative source will show that both words are correct
> in this context, I'll readily admit that I'm wrong and will remember it
> for the future. I didn't comment on a whim, it wasn't a matter of taste,
> it was as far as I know an actual incorrect usage of words, an error.

Go ahead and play that game again.

>
> > > > > > > > +  table_list->vers_skip_create= false;
> > > > > > > > +  ot_ctx->vers_create_count= 0;
> > > > > > > > +  action= Open_table_context::OT_REOPEN_TABLES;
> > > > > > > > +  table_arg= NULL;
> > > > > > > > +}
> > > > > > >
> > > > > > > I'm afraid I don't understand. All this business with
> > > > > > > vers_skip_create and vers_skip_auto_create, it wasn't in the
> > > > > > > previous version of the patch. So, I believe it was a fix
> > > > > > > for one of the MDEV bugs reported and fixed meanwhile.
> > > > > >
> > > > > > No, that was the multi-threaded case which worked good for me,
> > > > > > but suddenly I discovered it fails on some buildbot.
> > > > >
> > > > > Could you elaborate on what the problem was? Two threads trying
> > > > > to add the partition in parallel? I'd expect MDL_EXCLUSIVE to
> > > > > prevent that.
> > > >
> > > > MDL_EXCLUSIVE prevents parallel execution of
> > > > repair_from_failed_open(), but not sequential. So it can add
> > > > several partitions instead of 1, one after another.
> > >
> > > What's the sequence of events? One thread decides to add a
> > > partition, takes an MDL_EXCLUSIVE, the other thread decides to add a
> > > partition, waits for MDL_EXCLUSIVE, the first one finishes adding a
> > > partition, releases the lock, the second grabs it and adds a second
> > > partition?
> >
> > Right.
>
> Okay. Then, why a thread didn't check the number of partitions after
> acquiring MDL_EXCLUSIVE? Like with mutexes, if there's a mutex that
> protects a shared variable, you first acquire the mutex, then read the
> variable's value. Not the other way around.

Number of partitions is not a shared variable. part_info is kept in
TABLE instance. To get new value it must reacquire share, reparse
part_sql string. Then comparing with what? After releasing
MDL_SHARED_WRITE everything is gone: TABLE, TABLE_SHARE. You must
store somewhere old value, presumably in Open_table_context.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org


-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 8e79d168614: MDEV-25672 table alias from previous statement interferes later commands.

[Aleksey Midenkov]

Hi Sergei!

I'm sorry, what other I can say apart from commits descriptions? This
code fixes the roots of problems while you suggest to keep bugs
masked.

On Wed, Jun 2, 2021 at 3:23 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 02, Aleksey Midenkov wrote:
> > revision-id: 8e79d168614 (mariadb-10.2.31-989-g8e79d168614)
> > parent(s): 433dd490d33
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2021-05-31 16:20:38 +0300
> > message:
> >
> > MDEV-25672 table alias from previous statement interferes later commands.
>
> I'm sorry. I don't understand what all this new code for.
>
> To make sure that vcol's field->table_name is refreshed for every statement?
> What is vcol's field->table_name used for besides your CREATE TABLE check?
>
> If it's not used for anything then a much simpler fix would be narrow
> the check - it only needs to be run for new vcols in CREATE TABLE,
> disabling it for old vcols in ALTER TABLE will do the trick. Like
>
> -if (p.table_name.length && table_name)
> +if (!field && p.table_name.length && table_name)
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Wed, Jun 2, 2021 at 12:06 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 01, Aleksey Midenkov wrote:
> > > > +# Don't auto-create new partition on DELETE HISTORY:
> > > > +set timestamp= unix_timestamp('2000-01-01 00:00:00');
> > > > +create or replace table t (a int) with system versioning
> > > > +partition by system_time interval 1 hour auto;
> > > > +set timestamp= unix_timestamp('2000-01-01 10:00:00');
> > >
> > > perhaps, make it only 02:00:00 to avoid any chance that
> > > a partition won't be created because of max-auto-create limit.
> > > it's possible we put the limit at 10, but 2 is very unlikely.
> > >
> > > or does max-auto-create limit always result in a warning?
> >
> > What is max-auto-create limit and how do we make it 10? Did you mean
> > that condition:
> >
> >   if (*create_count == MAX_PARTITIONS - 2)
> >   {
> > my_error(ER_TOO_MANY_PARTITIONS_ERROR, MYF(0));
> > return true;
> >   }
>
> Yes, that's what I meant. Ok, there will be an error, good.
>
> > > > diff --git a/mysql-test/suite/versioning/r/partition.result 
> > > > b/mysql-test/suite/versioning/r/partition.result
> > > > --- a/mysql-test/suite/versioning/r/partition.result
> > > > +++ b/mysql-test/suite/versioning/r/partition.result
> > > > @@ -1236,27 +1270,826 @@ t1   CREATE TABLE `t1` (
> ...
> > > > +Warnings:
> > > > +Warning  4114Versioned table `test`.`t1`: last HISTORY 
> > > > partition (`p0`) is out of INTERVAL, need more HISTORY partitions
> > > > +select * from t1 for system_time as of '2000-01-01 01:00:00';
> > > > +x
> > > > +select * from t1 partition (p0) order by x;
> > > > +x
> > > > +440
> > > > +441
> > > > +alter table t1 add partition partitions 3;
> > > > +select * from t1 for system_time as of '2000-01-01 01:00:00';
> > > > +x
> > > > +441
> > > > +select * from t1 partition (p0) order by x;
> > > > +x
> > > > +440
> > >
> > > would be good to have EXPLAINs after the overflow and after the manual 
> > > fix.
> > > to show how ALTER changes what partition will be pruned
>
> What about this? Agree/disagree?

Agree.

>
> > > > diff --git a/mysql-test/suite/versioning/t/partition.test 
> > > > b/mysql-test/suite/versioning/t/partition.test
> > > > --- a/mysql-test/suite/versioning/t/partition.test
> > > > +++ b/mysql-test/suite/versioning/t/partition.test
> > > > @@ -1068,13 +1078,456 @@ alter table t1 add partition partitions 2;
> > > >  --replace_result $default_engine DEFAULT_ENGINE
> > > >  show create table t1;
> > > >  alter table t1 add partition partitions 8;
> > > > +drop table t1;
> > > > +
> > > > +--echo #
> > > > +--echo # End of 10.5 tests
> > > > +--echo #
> > > > +
> > > > +--echo #
> > > > +--echo # MDEV-17554 Auto-create new partition for system versioned 
> > > > tables with history partitioned by INTERVAL/LIMIT
> > > > +--echo #
> > > > +create or replace table t1 (x int) with system versioning
> > > > +partition by system_time limit 1 auto;
> > > > +--replace_result $default_engine DEFAULT_ENGINE
> > > > +show create table t1;
> > > > +
> > >
> > > in the previous version of this patch you had clarifying comments here, 
> > > like
> > >
> > > --echo # INSERT, INSERT .. SELECT don't auto-create partitions
> > >
> > > or
> > >
> > > --echo # Number of partitions goes from 3 to 5
> > >
> > > I kind of miss them now, they were helpful
> >
> > Yes, they were helpful as is. I removed them like I promised in the
> > previous thread. Though, Sergei, you are free to add your own comments
> > in the form you like. I'm really sorry, but nobody won't tell me what
> > words to use unless I am not able to be clear and bear the common
> > sense.
>
> okay
>
> > > > diff --git a/sql/ha_partition.h b/sql/ha_partition.h
> > > > --- a/sql/ha_partition.h
> > > > +++ b/sql/ha_partition.h
> > > > @@ -1607,7 +1607,7 @@ class ha_partition :public handler
> > > >  for (; part_id < part_id_end; ++part_id)
> > > >  {
> > > >handler *file= m_fi

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()

[Aleksey Midenkov]

Hi Sergei!

Users could benefit from this fix of MDEV-18734 since the last
changes. Do you really think the open questions left are so important
for this bug to be still not pushed?

On Tue, Jan 26, 2021 at 1:51 AM Aleksey Midenkov  wrote:
>
> Hi Sergei,
>
> On Tue, Dec 22, 2020 at 7:58 PM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey!
> >
> > On Nov 03, Aleksey Midenkov wrote:
> > >
> > > the updated patches are here:
> > >
> > > https://github.com/MariaDB/server/commits/5f6bcd6cd0145513974b0dfc5b2cefba28b72f1a
> >
> > I started writing my comments there, but realized that it'd be clearer
> > if I reply here, more context. That is, see below.
> >
> > One comment about the new test case, you have 16K lines in the result
> > file, are all 16K necessary? Can you do something like
> >
> >select x, left(b,10), left(v,10)
> >
> > ?
>
> Yes, these shorter result cases reproduce as well. Updated.
>
> >
> > And please, pretty please, don't use non-constant references as function
> > arguments.
>
> Why? With pointers we can pass NULL and we don't need NULL there, do we?
>
> >
> > > On Mon, Nov 2, 2020 at 11:33 PM Sergei Golubchik  wrote:
> > > >
> > > > > diff --git a/sql/field.cc b/sql/field.cc
> > > > > index bdaaecc20269..0fd40c979d2c 100644
> > > > > --- a/sql/field.cc
> > > > > +++ b/sql/field.cc
> > > > > @@ -8310,6 +8310,7 @@ int Field_blob::store(const char *from,uint 
> > > > > length,CHARSET_INFO *cs)
> > > > >copy_length= copier.well_formed_copy(field_charset,
> > > > > (char*) value.ptr(), 
> > > > > new_length,
> > > > > cs, from, length);
> > > > > +  value.length(copy_length);
> > > >
> > > > good! Could this be a distinct bug with its own test case?
> > >
> > > Do we need to spend time on that bureaucracy? There are real problems to
> > > solve... Tickets drain time from the management people and generally from
> > > anyone who sees them.
> >
> > I didn't mean opening an MDEV. Only writing a test case and extracting
> > it into a separate commit.
> >
> > If this is a genuine bug fix and you don't create a test case - how can
> > you be sure the fix will stay? It may disappear in a merge or a rebase,
> > manually or automatically. Or in a refactoring that "looks fine, all
> > tests pass". If you can create a test case - please, do.
>
> It is tested by federated.federated_partition test case. Why do we
> have to extract this side-fix to a separate commit? Looks anyway as a
> bureaucracy to me. The main bug is critical, the test case tests both
> bugs nicely. If the partitioning code is refactored, the test case
> still tests that side-fix.
>
> >
> > > > >Field_blob::store_length(copy_length);
> > > > >bmove(ptr+packlength,(uchar*) ,sizeof(char*));
> > > > >
> > > > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
> > > > > index 9d82df0235c7..7c658082d397 100644
> > > > > --- a/sql/ha_partition.cc
> > > > > +++ b/sql/ha_partition.cc
> > > > > @@ -5125,14 +5128,14 @@ bool 
> > > > > ha_partition::init_record_priority_queue()
> > > > >uint alloc_len;
> > > > >uint used_parts= bitmap_bits_set(_part_info->read_partitions);
> > > > >/* Allocate record buffer for each used partition. */
> > > > > -  m_priority_queue_rec_len= m_rec_length + PARTITION_BYTES_IN_POS;
> > > > > +  m_priority_queue_rec_len= m_rec_length + ORDERED_REC_OFFSET;
> > > > >if (!m_using_extended_keys)
> > > > >m_priority_queue_rec_len += m_file[0]->ref_length;
> > > > >alloc_len= used_parts * m_priority_queue_rec_len;
> > > > >/* Allocate a key for temporary use when setting up the scan. */
> > > > >alloc_len+= table_share->max_key_length;
> > > > >
> > > > > -  if (!(m_ordered_rec_buffer= (uchar*)my_malloc(alloc_len, 
> > > > > MYF(MY_WME
> > > > > +  if (!(m_ordered_rec_buffer= (uchar*) alloc_root(_ordered_root, 
> > > > > alloc_len)))
> > > >
> > > > I don't see why you need a memroot here. memroot is needed when you
> > > > allocate later at difeerent points in time some initially unpredict

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei!

I'm sending the reply but still didn't push the changes. Please expect
the changes pushed when I reassign MDEV-17554 to you.

On Sun, May 30, 2021 at 11:41 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Like last time, it's a review of the diff bc04ded2353 22158b3c05a,
> so not only commit 38a888da0f1.
>
> The main thing below - I didn't understand a couple of changes and asked
> for clarifications.
>
> On May 30, Aleksey Midenkov wrote:
> > revision-id: 38a888da0f1 (mariadb-10.5.2-653-g38a888da0f1)
> > parent(s): bc04ded2353
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2021-04-22 23:35:06 +0300
> > message:
> >
> > MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
>
>
> > diff --git a/mysql-test/suite/versioning/r/delete_history.result 
> > b/mysql-test/suite/versioning/r/delete_history.result
> > --- a/mysql-test/suite/versioning/r/delete_history.result
> > +++ b/mysql-test/suite/versioning/r/delete_history.result
> > @@ -154,3 +152,21 @@ select * from t1;
> >  a
> >  1
> >  drop table t1;
> > +#
> > +# MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
> > +#
> > +# Don't auto-create new partition on DELETE HISTORY:
> > +set timestamp= unix_timestamp('2000-01-01 00:00:00');
> > +create or replace table t (a int) with system versioning
> > +partition by system_time interval 1 hour auto;
> > +set timestamp= unix_timestamp('2000-01-01 10:00:00');
>
> perhaps, make it only 02:00:00 to avoid any chance that
> a partition won't be created because of max-auto-create limit.
> it's possible we put the limit at 10, but 2 is very unlikely.
>
> or does max-auto-create limit always result in a warning?
>

What is max-auto-create limit and how do we make it 10? Did you mean
that condition:

  if (*create_count == MAX_PARTITIONS - 2)
  {
my_error(ER_TOO_MANY_PARTITIONS_ERROR, MYF(0));
return true;
  }


> > +delete history from t;
> > +set timestamp= default;
> > +show create table t;
> > +TableCreate Table
> > +tCREATE TABLE `t` (
> > +  `a` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 
> > 00:00:00' AUTO
> > +PARTITIONS 2
> > +drop table t;
> > diff --git a/mysql-test/suite/versioning/r/partition.result 
> > b/mysql-test/suite/versioning/r/partition.result
> > --- a/mysql-test/suite/versioning/r/partition.result
> > +++ b/mysql-test/suite/versioning/r/partition.result
> > @@ -1236,27 +1270,826 @@ t1   CREATE TABLE `t1` (
> >   PARTITION `p5` HISTORY ENGINE = DEFAULT_ENGINE,
> >   PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE)
> >  alter table t1 add partition partitions 8;
> > +drop table t1;
> > +#
> > +# End of 10.5 tests
> > +#
> > +#
> > +# MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
> > +#
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time limit 1 auto;
> >  show create table t1;
> >  TableCreate Table
> >  t1   CREATE TABLE `t1` (
> >`x` int(11) DEFAULT NULL
> >  ) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > - PARTITION BY SYSTEM_TIME LIMIT 1
> > -(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE,
> > - PARTITION `p8` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO
> > +PARTITIONS 2
> > +insert into t1 values (1);
> > +create or replace table t2 (y int);
> > +insert into t2 values (2);
> > +insert into t1 select * from t2;
> > +insert into t2 select * from t1;
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO
> > +PARTITIONS 2
> > +# Too many partitions error
> > +set timestamp= unix_timestamp('2000-01-01 00:00:00');
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time interval 1 hour auto;
> > +set timestamp= unix_timestamp('2001-01-01 00:01:00');
> > +update t1 set x= x + 1;
> > +ERROR HY000: Too many partitions (including subpartitions) were defined
> > +# Partition overflow error and manu

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi, Sergei!

That "double standard" is an obstacle for users to achieve maximum
potential of software. Advanced users must know how internals work and
because this is not documented it is frequently hard work. I'm just
telling the downsides of the double concept. I know there are upsides
of course: less documentation, smooth learning curve. But now and then
developers make the dumbest from a user limiting his possibilities to
a real frustration. I just want you to see my point and I feel like
this happens every time with every software.

On Thu, May 27, 2021 at 12:41 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 27, Aleksey Midenkov wrote:
> > > On Apr 06, Aleksey Midenkov wrote:
>
> > > > To be able to specify them in INSERT command they must be at least
> > > > user-invisible. System-invisible fields are ignored.
> > >
> > > Sure, that's what @@system_versioning_insert_history would do.
> >
> > That's not about permission of inserting history which can be
> > controlled by @@secure_timestamp setting. But rather that's about
> > allowing to put system-invisible fields into INSERT command. You can
> > suggest a better name for it.
>
> yes, @@system_versioning_insert_history is not
> @@system_versioning_allow_permissisions_to_insert_history.
> So it's not about permissions, it's about "insert history [rows]" - with
> everything that's needed.
>
> > > No, this is internal implementation detail that can change and it
> > > should not leak into the UI. Like, sql_select.cc has a function
> > > called sub_select(), but we never tell users that what it does is
> > > "subselect", not in the documentation, not in error messages, never.
> > > Because it doesn't (it performs one step in a nested-loop join).
> >
> > Every concept should have a proper application. I don't think "double
> > design" in hiding system versioning fields adds any value. I can not
> > be sure about "subselect" design, but I suspect it will never be
> > replaced by anything else. In any case each and every solution should
> > be judged by whether it adds usability (i.e. ease of use) or not.
>
> Yes, and the intended and documented behavior, that you've implemented,
> is - either a user specifies row start/end fields explicitly, as the
> standard requires, and they exist as normal fields - or the user
> doesn't, and they don't.
>
> The internal implementation specifics don't necessarily have to leak
> into UI. For example, SEQUENCE objects - they are separate objects in
> the standard and mostly behave as such in MariaDB too. But internally
> they're implemented as some special kind of a table.
>
> Stored routines are also a completely separate kind of objects. Despite
> the fact that we store them as rows in mysql.proc table.
>
> So, if a user hasn't created row start/end columns - there are no row
> start/end columns. The server stores this information _somewhere_ and
> makes it available in SELECT queries as pseudocolumns.
>
> With @@system_versioning_insert_history these pseudocolumns are allowed
> in INSERT too.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Sergei,

sorry, I forgot to add that implementation with @@force_fields_visible
is cleaner and shorter. Also that helps to deprecate sysvers_show and
MDEV-16587 which I would greatly appreciate.

On Tue, Apr 27, 2021 at 4:07 PM Aleksey Midenkov  wrote:
>
> Hi Sergei!
>
> On Tue, Apr 6, 2021 at 3:29 PM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey!
> >
> > On Apr 06, Aleksey Midenkov wrote:
> > > > > But, since we need to specify implicit system fields we cannot
> > > > > avoid adding one more session variable. In my current iteration I
> > > > > made @@force_fields_visible which is more straightforward in what
> > > > > it does:
> > > >
> > > > I'm sorry, I don't understand.
> > > >
> > > > First, visibility is pretty much unrelated concept.
> > > > row_start/row_end can be visible or invisible, and they can be
> > > > writable or not writable - those are orthogonal concepts.
> > >
> > > To be able to specify them in INSERT command they must be at least
> > > user-invisible. System-invisible fields are ignored.
> >
> > Sure, that's what @@system_versioning_insert_history would do.
> >
>
> That's not about permission of inserting history which can be
> controlled by @@secure_timestamp setting. But rather that's about
> allowing to put system-invisible fields into INSERT command. You can
> suggest a better name for it.
>
> > > > And second, the name is wrong, there are no "fields" row_start and
> > > > row_end unless the user creates then explicitly. They are pieces of
> > > > metadata that every row has, something that Oracle, for example,
> > > > calls "pseudocolumns". Something like
> > > > @@system_versioning_row_start_row_end_visible would be more correct,
> > > > but ugly. In fact, I'd say that @@system_versioning_insert_history
> > > > was the best one.
> > >
> > > I think you are complicating things where complication is not needed.
> > > Pseudo- or not they are fields.
> >
> > No, this is internal implementation detail that can change and it should
> > not leak into the UI. Like, sql_select.cc has a function called
> > sub_select(), but we never tell users that what it does is "subselect",
> > not in the documentation, not in error messages, never. Because it
> > doesn't (it performs one step in a nested-loop join).
>
> Every concept should have a proper application. I don't think "double
> design" in hiding system versioning fields adds any value. I can not
> be sure about "subselect" design, but I suspect it will never be
> replaced by anything else. In any case each and every solution should
> be judged by whether it adds usability (i.e. ease of use) or not. I
> have a suspicion that the current design of hidden row_start/row_end
> adds more obstacles to a user than helps him.
>
> >
> > The fact that some internal enum value is named INVISIBLE_SYSTEM
> > is not something that should affect the user visible behavior.
> >
> > > Besides, the variable is not about system versioning. It can make any
> > > system-invisible fields visible.
> >
> > Which is incorrect. ROWNUM (MDEV-24089) is a pseudocolumn, and it cannot
> > be "visible" in the sense of @force_fields_visible. We don't support
> > ROWID, but if we would — it cannot be visible in the sense of
> > @force_fields_visible either. Basically this variable can only apply to
> > row_start/row_end pseudocolumns, despite its generic name.
>
> That is just the subject for a new feature request, isn't it? In any
> case, please suggest a different name. I can think of
> @@system_versioning_show_row_timestamps but this looks a bit long to
> me.
>
> >
> > Regards,
> > Sergei
> > VP of MariaDB Server Engineering
> > and secur...@mariadb.org
>
>
>
> --
> All the best,
>
> Aleksey Midenkov
> @midenok



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi Sergei!

On Tue, Apr 6, 2021 at 3:29 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 06, Aleksey Midenkov wrote:
> > > > But, since we need to specify implicit system fields we cannot
> > > > avoid adding one more session variable. In my current iteration I
> > > > made @@force_fields_visible which is more straightforward in what
> > > > it does:
> > >
> > > I'm sorry, I don't understand.
> > >
> > > First, visibility is pretty much unrelated concept.
> > > row_start/row_end can be visible or invisible, and they can be
> > > writable or not writable - those are orthogonal concepts.
> >
> > To be able to specify them in INSERT command they must be at least
> > user-invisible. System-invisible fields are ignored.
>
> Sure, that's what @@system_versioning_insert_history would do.
>

That's not about permission of inserting history which can be
controlled by @@secure_timestamp setting. But rather that's about
allowing to put system-invisible fields into INSERT command. You can
suggest a better name for it.

> > > And second, the name is wrong, there are no "fields" row_start and
> > > row_end unless the user creates then explicitly. They are pieces of
> > > metadata that every row has, something that Oracle, for example,
> > > calls "pseudocolumns". Something like
> > > @@system_versioning_row_start_row_end_visible would be more correct,
> > > but ugly. In fact, I'd say that @@system_versioning_insert_history
> > > was the best one.
> >
> > I think you are complicating things where complication is not needed.
> > Pseudo- or not they are fields.
>
> No, this is internal implementation detail that can change and it should
> not leak into the UI. Like, sql_select.cc has a function called
> sub_select(), but we never tell users that what it does is "subselect",
> not in the documentation, not in error messages, never. Because it
> doesn't (it performs one step in a nested-loop join).

Every concept should have a proper application. I don't think "double
design" in hiding system versioning fields adds any value. I can not
be sure about "subselect" design, but I suspect it will never be
replaced by anything else. In any case each and every solution should
be judged by whether it adds usability (i.e. ease of use) or not. I
have a suspicion that the current design of hidden row_start/row_end
adds more obstacles to a user than helps him.

>
> The fact that some internal enum value is named INVISIBLE_SYSTEM
> is not something that should affect the user visible behavior.
>
> > Besides, the variable is not about system versioning. It can make any
> > system-invisible fields visible.
>
> Which is incorrect. ROWNUM (MDEV-24089) is a pseudocolumn, and it cannot
> be "visible" in the sense of @force_fields_visible. We don't support
> ROWID, but if we would — it cannot be visible in the sense of
> @force_fields_visible either. Basically this variable can only apply to
> row_start/row_end pseudocolumns, despite its generic name.

That is just the subject for a new feature request, isn't it? In any
case, please suggest a different name. I can think of
@@system_versioning_show_row_timestamps but this looks a bit long to
me.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi, Sergei!

On Mon, Apr 12, 2021 at 8:46 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 12, Aleksey Midenkov wrote:
> > On Mon, Apr 12, 2021 at 3:14 PM Sergei Golubchik  wrote:
> > >
> > > On Apr 11, Aleksey Midenkov wrote:
> > > >
> > > > It was a conscious choice. Quantitative characteristic is implied.
> > > This isn't going anywhere.
> > Of course. And because of that I prefer to write shorter comments.
> > > I've asked Ian (KB technical writer) for his opinion and we'll do what
> > > he'll say.
> >
> > So he says it is not meaningless. I am and was upset by your
> > disagreement with my taste. If you are totally against the form I use
> > I'm going to remove the comments. But when we are talking about public
> > documents of course I'm going to make it better understandable, and he
> > is right regarding that particular place.
>
> I perceived it simply as incorrect, broken English. So I asked Ian's
> opinion. Also I believe we should avoid using different terminology
> internally and externally unless there's a good reason to do it.
> That's all, not a question of a taste.

I believe you are right in your motivation. But the emphasis in the
test is on increment: we are checking the number in PARTITIONS clause.
I removed all the comments regarding "increment" from the test.

>
> > > > > > I also have to revert DBUG_ASSERT() you suggested which fails
> > > > > > drop_table_force on winx64-debug and add error code condition 
> > > > > > instead:
> > > > > >
> > > > > > @@ -3268,12 +3271,12 @@ 
> > > > > > Open_table_context::recover_from_failed_open()
> > > > > > case OT_DISCOVER:
> > > > > > case OT_REPAIR:
> > > > > > case OT_ADD_HISTORY_PARTITION:
> > > > > > -  DBUG_ASSERT(!m_thd->get_stmt_da()->is_set());
> > > > > >   result= lock_table_names(m_thd, m_thd->lex->create_info, 
> > > > > > m_failed_table,
> > > > > >NULL, get_timeout(), 0);
> > > > > >   if (result)
> > > > > >   {
> > > > > > -if (m_action == OT_ADD_HISTORY_PARTITION)
> > > > > > +if (m_action == OT_ADD_HISTORY_PARTITION &&
> > > > > > +m_thd->get_stmt_da()->sql_errno() == 
> > > > > > ER_LOCK_WAIT_TIMEOUT)
> > > > > > {
> > > > > >   // MDEV-23642 Locking timeout caused by auto-creation 
> > > > > > affects original DML
> > > > > >   m_thd->clear_error();
> > > > >
> > > > > What happens on winx64-debug?
> > > >
> > > > Please look yourself at
> > > >
> > > > http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/24345/steps/test/logs/stdio
> > >
> > > Thanks. But either I don't know how to read it correctly or it
> > > doesn't have enough info.
> > >
> > > Could you explain what happens on this builder? What error do you
> > > see in stmt_da in recover_from_failed_open() that triggers the
> > > assert?
> >
> > Why are you asking me to do investigations on your hypotheses out of
> > the task scope? It was just a hypothesis after all!
>
> I meant that you do thd->clear_error(). It's non-discriminative, it
> clears all errors, no matter what was the error and where it was
> created.
>
> This is not always correct - you, probably, remember a bug with virtual
> columns when a vcol expression was parsed in the error handling branch
> and clear_error() removed the original error.
>
> Generally, one should avoid thd->clear_error() and push an error handler
> into the thd.
>
> But my hypothesis was that what you're doing here is safe, because there
> cannot be any error at this point. If my hypothesis is wrong, then you
> cannot use thd->clear_error(). Or, perhaps, my hypothesis is correct and
> there should be no error here, meaning some other code is wrong.
>
> I asked to be able to differentiate between these two possibilities. If
> my hypothesis is wrong, you cannot use thd->clear_error(). If my
> hypothesis is correct, some other code needs fixing.
>
> > Though you can
> > figure out error code (ER_NO_SUCH_TABLE) from the test case:
> >
> > create table t2(a int not null) engine=archive;
> > flush tables;
> > --error 1
> > --remove_file $DATA

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei,

On Mon, Apr 12, 2021 at 3:14 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 11, Aleksey Midenkov wrote:
> >
> > It was a conscious choice. Quantitative characteristic is implied.
>
> This isn't going anywhere.

Of course. And because of that I prefer to write shorter comments.

> I've asked Ian (KB technical writer) for his opinion and we'll do what
> he'll say.

So he says it is not meaningless. I am and was upset by your
disagreement with my taste. If you are totally against the form I use
I'm going to remove the comments. But when we are talking about public
documents of course I'm going to make it better understandable, and he
is right regarding that particular place.

>
> > > True, I know that my feeling of the language is not perfect, I'm not a
> > > native speaker. But it works both ways, you're not a native speaker
> > > either.
> > >
> > > So, if you don't want to change to correct English, there's no point in
> > > arguing, let's just ask a native speaker.
> > >
> > > > > I'd suggest to rewrite as
> > > > >
> > > > >if (ot_ctx->vers_create_count)
> > > > >  /*
> > > > >already tried to add a partition to this table and failed
> > > > >(because of e.g. lock conflict). Don't try again
> > > > >  */
> > > > >else if (table->vers_need_hist_part(thd, table_list))
> > > > > ... and here your code ...
> > > >
> > > > The comment is wrong. Added correct comment.
> > >
> > > The comment describes what I've seen in gdb, looking how
> > > ot_ctx->vers_create_count could be non-zero. It was your
> > >
> > >   update t1 set x= x + 122 where x = 1
> > >
> > > test case, "Locking timeout caused by auto-creation affects original DML"
> > > You try to auto-create, it fails because of the lock timeout, you
> > > clear_error(), the table is reopened. But because vers_create_count is
> > > not reset, it does not try to auto-create again, thus avoiding an
> > > infinite loop.
> > >
> > > What do you thing was wrong there?
> >
> > Okay, it is partly true. Failing case is the secondary one.
>
> May be, but it's the only case where ot_ctx->vers_create_count>0 there.
> So if there's a "primary" case too, your tests never trigger it.
> Add a test for it, please.

Sorry, my bad, vers_create_count was not passed to open_ltable(),
there is a different ot_ctx. Added your comment as is.

>
> > > > I also have to revert DBUG_ASSERT() you suggested which fails
> > > > drop_table_force on winx64-debug and add error code condition instead:
> > > >
> > > > @@ -3268,12 +3271,12 @@ Open_table_context::recover_from_failed_open()
> > > > case OT_DISCOVER:
> > > > case OT_REPAIR:
> > > > case OT_ADD_HISTORY_PARTITION:
> > > > -  DBUG_ASSERT(!m_thd->get_stmt_da()->is_set());
> > > >   result= lock_table_names(m_thd, m_thd->lex->create_info, 
> > > > m_failed_table,
> > > >NULL, get_timeout(), 0);
> > > >   if (result)
> > > >   {
> > > > -if (m_action == OT_ADD_HISTORY_PARTITION)
> > > > +if (m_action == OT_ADD_HISTORY_PARTITION &&
> > > > +m_thd->get_stmt_da()->sql_errno() == ER_LOCK_WAIT_TIMEOUT)
> > > > {
> > > >   // MDEV-23642 Locking timeout caused by auto-creation affects 
> > > > original DML
> > > >   m_thd->clear_error();
> > >
> > > What happens on winx64-debug?
> >
> > Please look yourself at
> >
> > http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/24345/steps/test/logs/stdio
>
> Thanks. But either I don't know how to read it correctly or it doesn't
> have enough info.
>
> Could you explain what happens on this builder? What error do you see in
> stmt_da in recover_from_failed_open() that triggers the assert?

Why are you asking me to do investigations on your hypotheses out of
the task scope? It was just a hypothesis after all! Though you can
figure out error code (ER_NO_SUCH_TABLE) from the test case:

create table t2(a int not null) engine=archive;
flush tables;
--error 1
--remove_file $DATADIR/test/t2.frm
select * from t2;
flush tables;
--remove_file $DATADIR/test/t2.ARZ
--error ER_NO_SUCH_TABLE
select * from t2;

There are several tests failing though...

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
--
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei,

On Sun, Apr 11, 2021 at 7:21 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 11, Aleksey Midenkov wrote:
> > On Sat, Apr 10, 2021 at 11:12 PM Sergei Golubchik  wrote:
> > > On Apr 09, Aleksey Midenkov wrote:
> > > > > >
> > > > > > --echo # Increment from 3 to 5
> > > > > > --echo # Increment from 3 to 6, manual names, LOCK TABLES
> > > > > > --echo # Multiple increments in single command
> > > > > >
> > > > > > Besides "increment" is correct because PARTITIONS number is
> > > > > > incremented.
> > > > >
> > > > > Sure.
> > > > > "Increment the number of partitions", this is fine.
> > > > > "Auto create partitions" is also fine.
> > > > > "Increment partitions" is meaningless.
> > > >
> > > > It is obvious from the context that we are talking about the number,
> > > > not partitions themselves. Treat "partitions" as PARTITIONS keyword
> > > > and the increment is attached to a number right next to it. That's
> > > > quite a sense, isn't it?
> > >
> > > No, I think it's a meaningless combination of words.
> > > But let's ask native speakers, shall we?
> >
> > The ability to imply things and make special terms is the freedom of
> > any language I suppose. Slang (or "special language") helps us to
> > describe technical entities more concisely.
>
> :)
> I feel it's just incorrect use of words and I shudder every time I see
> it.

It was a conscious choice. Quantitative characteristic is implied.

>
> True, I know that my feeling of the language is not perfect, I'm not a
> native speaker. But it works both ways, you're not a native speaker
> either.
>
> So, if you don't want to change to correct English, there's no point in
> arguing, let's just ask a native speaker.
>
> > > I'd suggest to rewrite as
> > >
> > >if (ot_ctx->vers_create_count)
> > >  /*
> > >already tried to add a partition to this table and failed
> > >(because of e.g. lock conflict). Don't try again
> > >  */
> > >else if (table->vers_need_hist_part(thd, table_list))
> > > ... and here your code ...
> >
> > The comment is wrong. Added correct comment.
>
> The comment describes what I've seen in gdb, looking how
> ot_ctx->vers_create_count could be non-zero. It was your
>
>   update t1 set x= x + 122 where x = 1
>
> test case, "Locking timeout caused by auto-creation affects original DML"
> You try to auto-create, it fails because of the lock timeout, you
> clear_error(), the table is reopened. But because vers_create_count is
> not reset, it does not try to auto-create again, thus avoiding an
> infinite loop.
>
> What do you thing was wrong there?

Okay, it is partly true. Failing case is the secondary one.

>
> > I also have to revert DBUG_ASSERT() you suggested which fails
> > drop_table_force on winx64-debug and add error code condition instead:
> >
> > @@ -3268,12 +3271,12 @@ Open_table_context::recover_from_failed_open()
> > case OT_DISCOVER:
> > case OT_REPAIR:
> > case OT_ADD_HISTORY_PARTITION:
> > -  DBUG_ASSERT(!m_thd->get_stmt_da()->is_set());
> >   result= lock_table_names(m_thd, m_thd->lex->create_info, 
> > m_failed_table,
> >        NULL, get_timeout(), 0);
> >   if (result)
> >   {
> > -if (m_action == OT_ADD_HISTORY_PARTITION)
> > +if (m_action == OT_ADD_HISTORY_PARTITION &&
> > +m_thd->get_stmt_da()->sql_errno() == ER_LOCK_WAIT_TIMEOUT)
> > {
> >   // MDEV-23642 Locking timeout caused by auto-creation affects 
> > original DML
> >   m_thd->clear_error();
>
> What happens on winx64-debug?

Please look yourself at

http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/24345/steps/test/logs/stdio

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei,

On Sat, Apr 10, 2021 at 11:12 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 09, Aleksey Midenkov wrote:
> > > >
> > > > --echo # Increment from 3 to 5
> > > > --echo # Increment from 3 to 6, manual names, LOCK TABLES
> > > > --echo # Multiple increments in single command
> > > >
> > > > Besides "increment" is correct because PARTITIONS number is incremented.
> > >
> > > Sure.
> > > "Increment the number of partitions", this is fine.
> > > "Auto create partitions" is also fine.
> > > "Increment partitions" is meaningless.
> >
> > It is obvious from the context that we are talking about the number,
> > not partitions themselves. Treat "partitions" as PARTITIONS keyword
> > and the increment is attached to a number right next to it. That's
> > quite a sense, isn't it?
>
> No, I think it's a meaningless combination of words.
> But let's ask native speakers, shall we?

The ability to imply things and make special terms is the freedom of
any language I suppose. Slang (or "special language") helps us to
describe technical entities more concisely.

>
> > > > > > +--echo # Here t2 is incremented too, but not updated
> > > > > > +set timestamp= unix_timestamp('2000-01-01 03:00:00');
> > > > > > +update t1, t2 set t1.x= 0 where t1.x< t2.y;
> > > > > > +--replace_result $default_engine DEFAULT_ENGINE
> > > > > > +show create table t1;
> > > > > > +# Multiupdate_prelocking_strategy::handle_end() is processed after 
> > > > > > table open.
> > > > > > +# For PS it is possible to skip unneeded auto-creation because the 
> > > > > > above happens at
> > > > > > +# prepare stage and auto-creation is done at execute stage.
> > > > > > +--replace_result $default_engine DEFAULT_ENGINE 'PARTITIONS 4' 
> > > > > > 'PARTITIONS ok' 'PARTITIONS 5' 'PARTITIONS ok'
> > > > >
> > > > > eh... I don't think this is really "ok".
> > > > > As far as I remember, Multiupdate_prelocking_strategy knows what
> > > > > tables should be opened for writing and what for reading. Why
> > > > > would a new partition be created for t2?
> > > >
> > > > It knows this after tables are opened. Look at handle_end(),
> > > > specifically mysql_handle_derived(), handle_derived(),
> > > > setup_fields_with_no_wrap() and check_fields(). I believe all these
> > > > calls are required to get proper get_table_map().
> > > >
> > > > To get this working properly there must be 2-staged open tables,
> > > > something like PS does.
> > >
> > > You mean, a new partition is created for t2 in normal mode and not
> > > created in --ps?
> >
> > Yes.
> >
> > > What if you'll also add `&& table_list->updating` to your condition in
> > > TABLE::vers_need_hist_part() ?
> >
> > If `Multiupdate_prelocking_strategy::handle_end()` was not yet
> > executed how `updating` can help? No, it doesn't help.
>
> Okay. I think I know how to fix it, but let's do it after the main thing
> is pushed.
>
> Users are rather sensitive to cases when a non-updatable table in
> multi-update gets treated like updatable, we've had a bunch of bug
> reports about it over the years. Wrong privileges (UPDATE privilege
> should not be checked), wrong locks (the table should never be
> write-locked, a concurrent write lock should not block multi-update),
> triggers (should not be opened, so any problems with triggers, like
> privileges or missing tables, should not affect multi-update), and so on.
>
> So I suppose we'll have to fix this one too. This is how it could be
> done - vers_need_hist_part should check for table_list->updating. It's
> always FALSE in multi-update, so it won't be auto-creating new
> partitions at all for multi-update. But! after handle_end() (or, better,
> in handle_end()) we'll have another pass over tables and will implement
> a fallback-and-retry from handle_end().
>
> Let's do it as a separate commit after the feature is pushed.
>
> > > > > > @@ -2032,6 +2083,20 @@ bool open_table(THD *thd, TABLE_LIST 
> > > > > > *table_list, Open_table_context *ot_ctx)
> > > > > >  tc_add_table(thd, table);
> > > > > >}
> > > > > >
> > > > > > +  if (!ot_ctx->vers_creat

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei,

On Thu, Apr 8, 2021 at 3:30 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 07, Aleksey Midenkov wrote:
> > > > +
> > > > +--echo # INSERT, INSERT .. SELECT don't increment partitions
> > >
> > > it's not really "increment", better say "don't auto-create"
> >
> > Actually I like "increment" more. "Auto-create" overcomplicates phrases:
> >
> > --echo # Increment from 3 to 5
> > --echo # Increment from 3 to 6, manual names, LOCK TABLES
> > --echo # Multiple increments in single command
> >
> > Besides "increment" is correct because PARTITIONS number is incremented.
>
> Sure.
> "Increment the number of partitions", this is fine.
> "Auto create partitions" is also fine.
> "Increment partitions" is meaningless.

It is obvious from the context that we are talking about the number,
not partitions themselves. Treat "partitions" as PARTITIONS keyword
and the increment is attached to a number right next to it. That's
quite a sense, isn't it?

>
> > > > +insert into t1 values (1);
> > > > +
> > > > +create or replace table t2 (y int);
> > > > +insert into t2 values (2);
> > > > +
> > > > +insert into t1 select * from t2;
> > > > +insert into t2 select * from t1;
> > >
> > > why do you need a t2 table in this test?
> >
> > t1 is not incremented in case of
> >
> > insert into t2 select * from t1;
>
> add a comment please

Added.

>
> > > > +--echo # Here t2 is incremented too, but not updated
> > > > +set timestamp= unix_timestamp('2000-01-01 03:00:00');
> > > > +update t1, t2 set t1.x= 0 where t1.x< t2.y;
> > > > +--replace_result $default_engine DEFAULT_ENGINE
> > > > +show create table t1;
> > > > +# Multiupdate_prelocking_strategy::handle_end() is processed after 
> > > > table open.
> > > > +# For PS it is possible to skip unneeded auto-creation because the 
> > > > above happens at
> > > > +# prepare stage and auto-creation is done at execute stage.
> > > > +--replace_result $default_engine DEFAULT_ENGINE 'PARTITIONS 4' 
> > > > 'PARTITIONS ok' 'PARTITIONS 5' 'PARTITIONS ok'
> > >
> > > eh... I don't think this is really "ok".
> > > As far as I remember, Multiupdate_prelocking_strategy knows what tables
> > > should be opened for writing and what for reading. Why would a new 
> > > partition
> > > be created for t2?
> >
> > It knows this after tables are opened. Look at handle_end(),
> > specifically mysql_handle_derived(), handle_derived(),
> > setup_fields_with_no_wrap() and check_fields(). I believe all these
> > calls are required to get proper get_table_map().
> >
> > To get this working properly there must be 2-staged open tables,
> > something like PS does.
>
> You mean, a new partition is created for t2 in normal mode and not
> created in --ps?

Yes.

>
> What if you'll also add `&& table_list->updating` to your condition in
> TABLE::vers_need_hist_part() ?

If `Multiupdate_prelocking_strategy::handle_end()` was not yet
executed how `updating` can help? No, it doesn't help.

>
> > > > diff --git a/mysql-test/suite/versioning/r/partition.result 
> > > > b/mysql-test/suite/versioning/r/partition.result
> > > > index 2a277b1c2ea..8369b40d98c 100644
> > > > --- a/mysql-test/suite/versioning/r/partition.result
> > > > +++ b/mysql-test/suite/versioning/r/partition.result
> > > > @@ -1236,27 +1270,752 @@ t1   CREATE TABLE `t1` (
> > > ...
> > > > +set timestamp= unix_timestamp('2000-01-01 03:00:00');
> > > > +affected rows: 0
> > > > +lock tables t1 write;
> > > > +affected rows: 0
> > > > +execute s;
> > > > +affected rows: 1
> > > > +info: Rows matched: 1  Changed: 1  Inserted: 1  Warnings: 0
> > > > +execute s;
> > > > +affected rows: 1
> > > > +info: Rows matched: 1  Changed: 1  Inserted: 1  Warnings: 0
> > > > +show create table t1;
> > > > +TableCreate Table
> > > > +t1   CREATE TABLE `t1` (
> > > > +  `x` int(11) DEFAULT NULL
> > > > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > > > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 
> > > > 00:00:00' AUTO
> > > > +PARTITIONS 6
> > >
> > > wh

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei,

On Sun, Apr 4, 2021 at 2:48 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Note, it's a review of the `git diff 82e44d60d1e 8175ce8a5f1`
> so it's not only commit c4de76aeff8
>
> On Apr 04, Aleksey Midenkov wrote:
> > revision-id: c4de76aeff8 (mariadb-10.5.2-540-gc4de76aeff8)
> > parent(s): 82e44d60d1e
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2021-03-31 21:17:55 +0300
> > message:
> >
> > MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
>
> Overall it's very good! I have few minor questions/comments, see below.
> But it's almost done, I think.
>
> > diff --git a/mysql-test/suite/versioning/r/delete_history.result 
> > b/mysql-test/suite/versioning/r/delete_history.result
> > index cb865a835b3..90c9e4777bb 100644
> > --- a/mysql-test/suite/versioning/r/delete_history.result
> > +++ b/mysql-test/suite/versioning/r/delete_history.result
> > @@ -154,3 +152,18 @@ select * from t1;
> >  a
> >  1
> >  drop table t1;
> > +#
> > +# MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
> > +#
> > +# Don't auto-create new partition on DELETE HISTORY:
> > +create or replace table t (a int) with system versioning
> > +partition by system_time limit 1000 auto;
> > +delete history from t;
> > +show create table t;
> > +TableCreate Table
> > +tCREATE TABLE `t` (
> > +  `a` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO
> > +PARTITIONS 2
>
> Hmm, and if DELETE HISTORY would auto-create new partitions,
> what output would one expect here?
>
> I mean, how one can see whether the test result is correct or wrong?

By PARTITIONS value. Stale test case, fixed.

>
> > +drop table t;
> > diff --git a/mysql-test/suite/versioning/t/partition.test 
> > b/mysql-test/suite/versioning/t/partition.test
> > index 445f5844630..57b80ca0b47 100644
> > --- a/mysql-test/suite/versioning/t/partition.test
> > +++ b/mysql-test/suite/versioning/t/partition.test
> > @@ -1068,11 +1078,412 @@ alter table t1 add partition partitions 2;
> >  --replace_result $default_engine DEFAULT_ENGINE
> >  show create table t1;
> >  alter table t1 add partition partitions 8;
> > +drop table t1;
> > +
> > +--echo #
> > +--echo # MDEV-17554 Auto-create new partition for system versioned tables 
> > with history partitioned by INTERVAL/LIMIT
> > +--echo #
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time limit 1 auto;
> > +--replace_result $default_engine DEFAULT_ENGINE
> > +show create table t1;
> > +
> > +--echo # INSERT, INSERT .. SELECT don't increment partitions
>
> it's not really "increment", better say "don't auto-create"

Actually I like "increment" more. "Auto-create" overcomplicates phrases:

--echo # Increment from 3 to 5
--echo # Increment from 3 to 6, manual names, LOCK TABLES
--echo # Multiple increments in single command

Besides "increment" is correct because PARTITIONS number is incremented.

>
> > +insert into t1 values (1);
> > +
> > +create or replace table t2 (y int);
> > +insert into t2 values (2);
> > +
> > +insert into t1 select * from t2;
> > +insert into t2 select * from t1;
>
> why do you need a t2 table in this test?

t1 is not incremented in case of

insert into t2 select * from t1;

>
> > +--replace_result $default_engine DEFAULT_ENGINE
> > +show create table t1;
> > +
> > +--echo # Too many partitions error
> > +set timestamp= unix_timestamp('2000-01-01 00:00:00');
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time interval 1 hour auto;
> > +set timestamp= unix_timestamp('2001-01-01 00:01:00');
> > +--error ER_TOO_MANY_PARTITIONS_ERROR
> > +update t1 set x= x + 1;
> > +
> > +--enable_info
> > +--echo # Increment from 3 to 5
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time interval 3600 second
> > +starts '2000-01-01 00:00:00' auto partitions 3;
> > +
> > +insert into t1 values (1);
> > +--replace_result $default_engine DEFAULT_ENGINE
> > +show create table t1;
> > +
> > +set timestamp= unix_timestamp('2000-01-01 02:00:00');
> > +update t1 set x= x + 1;
> > +--replace_result $default_engine DEFAULT_ENGINE
>

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi Sergei,

On Mon, Apr 5, 2021 at 8:04 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Feb 26, Aleksey Midenkov wrote:
> > > > >
> > > > > In other words, if one can create an arbitrary history by
> > > > > manipulating @@timestamp variable,
> > > > > @@system_versioning_insert_history allows to do it more
> > > > > conveniently. But if one cannot - he cannot.
> > > >
> > > > Then do we need additional setting @@system_versioning_insert_history?
> > > > Iff one can manipulate history via @@timestamp variable let him set
> > > > row_start/row_end columns.
> > >
> > > Sure, that's possible.
> > >
> > > I just thought @@system_versioning_insert_history could be like an extra
> > > safety (not security) measure. To prevent history from being modified
> > > unintentionally.
> >
> > Well, unless one specified row_start/row_end explicitly he is safe.
> >
> > But, since we need to specify implicit system fields we cannot avoid
> > adding one more session variable. In my current iteration I made
> > @@force_fields_visible which is more straightforward in what it does:
>
> I'm sorry, I don't understand.
>
> First, visibility is pretty much unrelated concept. row_start/row_end
> can be visible or invisible, and they can be writable or not writable -
> those are orthogonal concepts.

To be able to specify them in INSERT command they must be at least
user-invisible. System-invisible fields are ignored.

>
> And second, the name is wrong, there are no "fields" row_start and
> row_end unless the user creates then explicitly. They are pieces of
> metadata that every row has, something that Oracle, for example, calls
> "pseudocolumns". Something like @@system_versioning_row_start_row_end_visible
> would be more correct, but ugly. In fact, I'd say that
> @@system_versioning_insert_history was the best one.

I think you are complicating things where complication is not needed.
Pseudo- or not they are fields. Besides, the variable is not about
system versioning. It can make any system-invisible fields visible.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 9bf4b92cbc5: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi, Sergei!

On Wed, Mar 31, 2021 at 5:52 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Dec 01, Sergei Golubchik wrote:
> > Hi, Aleksey!
> >
> > just one comment below, no comments on all your other replies.
> >
> > The issue is in review, shall I take another look or it's on you now?
>
> I think I've never got a reply to this question.
> Shall I take another look at the bb-10.6-midenok-MDEV-17554 branch or
> not yet?

I've updated the branch on top of the latest 10.6. Please go ahead and
take a look.

>
> > On Sep 01, Aleksey Midenkov wrote:
> >
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Sergei,

On Thu, Feb 25, 2021 at 11:09 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Feb 25, Aleksey Midenkov wrote:
> > > On Nov 24, Aleksey Midenkov wrote:
> > > > 1. Add server variable system_versioning_insert_history which will
> > > > allow INSERT history rows.
> > > > 2. If secure_timestamp is YES or REPLICATION,
> > > > system_versioning_insert_history does not have effect. If
> > > > secure_timestamp is SUPER, system_versioning_insert_history
> > > > requires special privilege (same as for setting current
> > > > timestamp).
> > >
> > > Yes. It is correct, but could be expressed simpler:
> > >
> > >   when @@system_versioning_insert_history=TRUE, one can directly insert
> > >   into row_start/row_end columns iff one can set @@timestamp variable.
> > >
> > > In other words, if one can create an arbitrary history by
> > > manipulating @@timestamp variable,
> > > @@system_versioning_insert_history allows to do it more
> > > conveniently. But if one cannot - he cannot.
> >
> > Then do we need additional setting @@system_versioning_insert_history?
> > Iff one can manipulate history via @@timestamp variable let him set
> > row_start/row_end columns.
>
> Sure, that's possible.
>
> I just thought @@system_versioning_insert_history could be like an extra
> safety (not security) measure. To prevent history from being modified
> unintentionally.
>

Well, unless one specified row_start/row_end explicitly he is safe.

But, since we need to specify implicit system fields we cannot avoid
adding one more session variable. In my current iteration I made
@@force_fields_visible which is more straightforward in what it does:

--- a/sql/table.cc
+++ b/sql/table.cc
@@ -4024,6 +4024,9 @@ enum open_frm_error open_table_from_share(THD
*thd, TABLE_SHARE *share,
  {
if (!((*field_ptr)= share->field[i]->clone(>mem_root, outparam)))
  goto err;
+if (thd->variables.force_fields_visible &&
+(*field_ptr)->invisible <= INVISIBLE_SYSTEM)
+  (*field_ptr)->invisible= VISIBLE;
  }
  (*field_ptr)= 0;  // End marker

This variable is more powerful as it affects any SQL command and I
hope this can be helpful. Please review the updated task in
bb-10.6-midenok

-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi, Sergei!

On Tue, Dec 1, 2020 at 7:54 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Nov 24, Aleksey Midenkov wrote:
> >
> > And mysqldump should take care of setting system_versioning_insert_history
> > sysvar. Right? Is everything else in the task description OK?
>
> Right.
>
> > 1. Add server variable system_versioning_insert_history which will allow
> > INSERT history rows.
> > 2. If secure_timestamp is YES or REPLICATION,
> > system_versioning_insert_history does not have effect. If secure_timestamp
> > is SUPER, system_versioning_insert_history requires special privilege (same
> > as for setting current timestamp).
>
> Yes. It is correct, but could be expressed simpler:
>
>   when @@system_versioning_insert_history=TRUE, one can directly insert
>   into row_start/row_end columns iff one can set @@timestamp variable.
>
> In other words, if one can create an arbitrary history by manipulating
> @@timestamp variable, @@system_versioning_insert_history allows to do it
> more conveniently. But if one cannot - he cannot.

Then do we need additional setting @@system_versioning_insert_history?
Iff one can manipulate history via @@timestamp variable let him set
row_start/row_end columns.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org


-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()

[Aleksey Midenkov]

Hi Sergei,

On Tue, Dec 22, 2020 at 7:58 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Nov 03, Aleksey Midenkov wrote:
> >
> > the updated patches are here:
> >
> > https://github.com/MariaDB/server/commits/5f6bcd6cd0145513974b0dfc5b2cefba28b72f1a
>
> I started writing my comments there, but realized that it'd be clearer
> if I reply here, more context. That is, see below.
>
> One comment about the new test case, you have 16K lines in the result
> file, are all 16K necessary? Can you do something like
>
>select x, left(b,10), left(v,10)
>
> ?

Yes, these shorter result cases reproduce as well. Updated.

>
> And please, pretty please, don't use non-constant references as function
> arguments.

Why? With pointers we can pass NULL and we don't need NULL there, do we?

>
> > On Mon, Nov 2, 2020 at 11:33 PM Sergei Golubchik  wrote:
> > >
> > > > diff --git a/sql/field.cc b/sql/field.cc
> > > > index bdaaecc20269..0fd40c979d2c 100644
> > > > --- a/sql/field.cc
> > > > +++ b/sql/field.cc
> > > > @@ -8310,6 +8310,7 @@ int Field_blob::store(const char *from,uint 
> > > > length,CHARSET_INFO *cs)
> > > >copy_length= copier.well_formed_copy(field_charset,
> > > > (char*) value.ptr(), new_length,
> > > > cs, from, length);
> > > > +  value.length(copy_length);
> > >
> > > good! Could this be a distinct bug with its own test case?
> >
> > Do we need to spend time on that bureaucracy? There are real problems to
> > solve... Tickets drain time from the management people and generally from
> > anyone who sees them.
>
> I didn't mean opening an MDEV. Only writing a test case and extracting
> it into a separate commit.
>
> If this is a genuine bug fix and you don't create a test case - how can
> you be sure the fix will stay? It may disappear in a merge or a rebase,
> manually or automatically. Or in a refactoring that "looks fine, all
> tests pass". If you can create a test case - please, do.

It is tested by federated.federated_partition test case. Why do we
have to extract this side-fix to a separate commit? Looks anyway as a
bureaucracy to me. The main bug is critical, the test case tests both
bugs nicely. If the partitioning code is refactored, the test case
still tests that side-fix.

>
> > > >Field_blob::store_length(copy_length);
> > > >bmove(ptr+packlength,(uchar*) ,sizeof(char*));
> > > >
> > > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
> > > > index 9d82df0235c7..7c658082d397 100644
> > > > --- a/sql/ha_partition.cc
> > > > +++ b/sql/ha_partition.cc
> > > > @@ -5125,14 +5128,14 @@ bool ha_partition::init_record_priority_queue()
> > > >uint alloc_len;
> > > >uint used_parts= bitmap_bits_set(_part_info->read_partitions);
> > > >/* Allocate record buffer for each used partition. */
> > > > -  m_priority_queue_rec_len= m_rec_length + PARTITION_BYTES_IN_POS;
> > > > +  m_priority_queue_rec_len= m_rec_length + ORDERED_REC_OFFSET;
> > > >if (!m_using_extended_keys)
> > > >m_priority_queue_rec_len += m_file[0]->ref_length;
> > > >alloc_len= used_parts * m_priority_queue_rec_len;
> > > >/* Allocate a key for temporary use when setting up the scan. */
> > > >alloc_len+= table_share->max_key_length;
> > > >
> > > > -  if (!(m_ordered_rec_buffer= (uchar*)my_malloc(alloc_len, 
> > > > MYF(MY_WME
> > > > +  if (!(m_ordered_rec_buffer= (uchar*) alloc_root(_ordered_root, 
> > > > alloc_len)))
> > >
> > > I don't see why you need a memroot here. memroot is needed when you
> > > allocate later at difeerent points in time some initially unpredictable
> > > amount of memory that has the same lifetime and needs to be
> > > freed all at once.
> >
> > I know... This is just another convenient way of allocation with presumably
> > minimized number of mallocs (probably pre_alloc_size should be tweaked to a
> > better value).
>
> memroot wasn't created for that. There are approaches with much less
> overhead

Using memory pools like that is a nice design pattern for non-critical
sections. At least it is nicer than manual pointer management. I
believe I've seen the similar pattern in InnoDB with its memory pool
mem_heap_t even without preallocation! It would be nice to have a
special class that preallocates and manages pointers, b

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hi Sergei!

On Tue, Sep 22, 2020 at 8:31 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> Wow, this comes so late, sorry! :(
>
> But it's fixVersion=10.6 and on the top my list, so here you are.
>
> On May 07, Aleksey Midenkov wrote:
> > >
> > > The conclusion, I believe, is that system_versioning_modify_history
> > > should allow INSERTs when secure_timestamp=NO, and it should allow
> > > UPDATE/DELETE only for a SUPER user when secure_timestamp=NO or SUPER.
> >
> > I don't see a reason to argue on that. The only thing that is not
> > clear, why we don't allow INSERTs when secure_timestamp=SUPER?
>
> Sure, I meant back then that INSERT/UPDATE/DELETE would be allowed when
> secure_timestamp=SUPER.
>
> But now I think we shouldn't really allow UPDATE/DELETE at all.
>
> I wrote earlier that "Now, with a SUPER privilege and
> secure_timestamp=NO or SUPER, one can use the BINLOG command and truly
> edit the history arbitrarily, by faking row events."
>
> Which is true, but we want to change it.
> In https://jira.mariadb.org/browse/MDEV-18432.
>
> So, it's looks reasonable to allow only INSERTs - they can fake new
> history, which is needed for mysqldump, but at least there will be no
> command to selectively delete or modify history.
>
> > > The second thing I don't like at all, is when a table is created like
> > >
> > >   CREATE TABLE t1 (a int) WITH SYSTEM VERSIONING
> > >
> > > with row_start/row_end implicit. You don't have it in the test, but
> > > anyway one should be able to load history into such a table, while
> > > the table does not have row_start and row_end columns. From the user
> > > point of view these columns don't exist, they're pseudo-columns,
> > > like ROWID. They just cannot be insertable-into, conceptually. But a
> > > user will want to restore the history, right? I don't have a
> > > solution for this yet :( Any ideas?
> >
> > We don't have to follow the conception if it doesn't help us. Since we
> > have physical row_start/row_end, we don't have to pretend they don't
> > exist. Who will win from that?
>
> Yes, you're right.
>
> Then for the purpose of mysqldump they should be insertable-into. And
> mysqldump should only care to list them explicitly in the column list,
> like with any other insivible columns. This looks easy.
>

And mysqldump should take care of setting system_versioning_insert_history
sysvar. Right? Is everything else in the task description OK?

1. Add server variable system_versioning_insert_history which will allow
INSERT history rows.
2. If secure_timestamp is YES or REPLICATION,
system_versioning_insert_history does not have effect. If secure_timestamp
is SUPER, system_versioning_insert_history requires special privilege (same
as for setting current timestamp).


>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()

[Aleksey Midenkov]

Hi Sergei!

The updated patches are here:

https://github.com/MariaDB/server/commits/2c2e79487a9189e02fce8f884a9ab2b42b9aa333


On Tue, Nov 3, 2020 at 8:28 PM Aleksey Midenkov  wrote:
>
> Sergei,
>
> the updated patches are here:
>
>
https://github.com/MariaDB/server/commits/5f6bcd6cd0145513974b0dfc5b2cefba28b72f1a
>
> On Mon, Nov 2, 2020 at 11:33 PM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey!
> >
> > Don't be confused, the commit header and the comment come from your last
> > commit, but the diff below includes all three commits that mention
MDEV-18734.
> >
...
>
> > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc
> > > index 9d82df0235c7..7c658082d397 100644
> > > --- a/sql/ha_partition.cc
> > > +++ b/sql/ha_partition.cc
> > > @@ -5125,14 +5128,14 @@ bool
ha_partition::init_record_priority_queue()
> > >uint alloc_len;
> > >uint used_parts= bitmap_bits_set(_part_info->read_partitions);
> > >/* Allocate record buffer for each used partition. */
> > > -  m_priority_queue_rec_len= m_rec_length + PARTITION_BYTES_IN_POS;
> > > +  m_priority_queue_rec_len= m_rec_length + ORDERED_REC_OFFSET;
> > >if (!m_using_extended_keys)
> > >m_priority_queue_rec_len += m_file[0]->ref_length;
> > >alloc_len= used_parts * m_priority_queue_rec_len;
> > >/* Allocate a key for temporary use when setting up the scan. */
> > >alloc_len+= table_share->max_key_length;
> > >
> > > -  if (!(m_ordered_rec_buffer= (uchar*)my_malloc(alloc_len,
MYF(MY_WME
> > > +  if (!(m_ordered_rec_buffer= (uchar*) alloc_root(_ordered_root,
alloc_len)))
> >
> > I don't see why you need a memroot here. memroot is needed when you
> > allocate later at difeerent points in time some initially unpredictable
> > amount of memory that has the same lifetime and needs to be
> > freed all at once.
>
> I know... This is just another convenient way of allocation with
presumably minimized number of mallocs (probably pre_alloc_size should be
tweaked to a better value).
>
> >
> > But here it seems that you only allocate once. You could just make
> >
> >m_priority_queue_rec_len = m_rec_length + PARTITION_BYTES_IN_POS +
> >   table->s->blob_fields * sizeof(Ordered_blob_storage)
>
> I don't want this data serialized. It is inconvenient to read the dump of
the record. Why we should make it harder to code, read the code and debug
while we can make it easier? If it is only for guaranteed 1 malloc, I don't
think it's worth it.
>

Added another patch that does 1 malloc in m_ordered_root.


--
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()

[Aleksey Midenkov]

Sergei,

the updated patches are here:

https://github.com/MariaDB/server/commits/5f6bcd6cd0145513974b0dfc5b2cefba28b72f1a

On Mon, Nov 2, 2020 at 11:33 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Don't be confused, the commit header and the comment come from your last
> commit, but the diff below includes all three commits that mention
MDEV-18734.
>
> On Nov 02, Aleksey Midenkov wrote:
> > revision-id: 2f9bbf392072 (mariadb-10.2.31-543-g2f9bbf392072)
> > parent(s): 1f4960e3f2ac
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2020-11-02 14:26:04 +0300
> > message:
> >
> > MDEV-18734 optimization by taking ownership via String::swap()
> >
> > ha_partition stores records in array of m_ordered_rec_buffer and uses
> > it for prio queue in ordered index scan. When the records are restored
> > from the array the blob buffers may be already freed or
> > rewritten. This can happen when blob buffers are cached in
> > Field_blob::value or Field_blob::read_value.
> >
> > Previous solution worked via copying blob buffer into
> > mem_root-allocated buffer.  That was not optimal as it requires more
> > memory and processing time.
> >
> > This patch avoids memory copy by taking temporary ownership of cached
> > blob buffers via String::swap().  It also fixes problem when
> > value.str_length is wrong after Field_blob::store().
>
>
> > diff --git a/mysql-test/suite/federated/federated_partition.test
b/mysql-test/suite/federated/federated_partition.test
> > index ef1e27ec5054..33ee025442f4 100644
> > --- a/mysql-test/suite/federated/federated_partition.test
> > +++ b/mysql-test/suite/federated/federated_partition.test
> > @@ -50,4 +50,30 @@ drop table federated.t1_2;
> >
> >  --echo End of 5.1 tests
> >
> > +--echo #
> > +--echo # MDEV-18734 ASAN heap-use-after-free upon sorting by blob
column from partitioned table
> > +--echo #
> > +connection slave;
> > +use federated;
> > +create table t1_1 (x int, b text, key(x));
> > +create table t1_2 (x int, b text, key(x));
> > +connection master;
> > +--replace_result $SLAVE_MYPORT SLAVE_PORT
> > +eval create table t1 (x int, b text, key(x)) engine=federated
> > +  partition by range columns (x) (
> > +  partition p1 values less than (40)
connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_1',
> > +  partition pn values less than (maxvalue)
connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2'
> > +);
> > +insert t1 values (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7,
7), (8, 8);
> > +insert t1 select x + 8, x + 8 from t1;
> > +insert t1 select x + 16, x + 16 from t1;
> > +insert t1 select x + 49, repeat(x + 49, 100) from t1;
> > +--echo # This produces wrong result before MDEV-17573
> > +select * from t1;
>
> What do you mean, simple `select * from t1` produces incorrect result?

Removed that garbage line.

>
> > +flush tables;
> > +select x, b from t1 where x > 30 and x < 60 order by b;
>
> Is this ASAN-only test?

No, it produces wrong result in 10.2 for me.

>
> > +drop table t1;
> > +connection slave;
> > +drop table t1_1, t1_2;
> > +
> >  source include/federated_cleanup.inc;
> > diff --git a/mysql-test/suite/vcol/t/partition.test
b/mysql-test/suite/vcol/t/partition.test
> > index 889724fb1c5c..f10ee0491ccc 100644
> > --- a/mysql-test/suite/vcol/t/partition.test
> > +++ b/mysql-test/suite/vcol/t/partition.test
> > @@ -30,3 +30,28 @@ subpartition by hash(v) subpartitions 3 (
> >  insert t1 set i= 0;
> >  set statement sql_mode= '' for update t1 set i= 1, v= 2;
> >  drop table t1;
> > +
> > +--echo #
> > +--echo # MDEV-18734 ASAN heap-use-after-free in
my_strnxfrm_simple_internal upon update on versioned partitioned table
> > +--echo #
> > +--echo # Cover queue_fix() in ha_partition::handle_ordered_index_scan()
> > +create table t1 (
> > +x int auto_increment primary key,
> > +b text, v mediumtext as (b) virtual,
> > +index (v(10))
> > +) partition by range columns (x) (
> > +partition p1 values less than (3),
> > +partition p2 values less than (6),
> > +partition pn values less than (maxvalue));
> > +insert into t1 (b) values ('q'), ('a'), ('b');
> > +update t1 set b= 'bar' where v > 'a';
> > +drop table t1;
>
> This test didn't fail for me in an ASAN build
> on the vanilla 10.5 without your fix.
> That is, I'm not sure it actually tests the fix.

It failed on 10.3 for me some time ago. Ok, replaced that with the bigger
version which fails on n

Re: [Maria-developers] 0cd5377e9ff: MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table

[Aleksey Midenkov]

Sergei,

On Sat, Oct 31, 2020 at 12:54 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Oct 25, Aleksey Midenkov wrote:
> > On Wed, Oct 21, 2020 at 4:19 PM Sergei Golubchik 
wrote:
> > > On Oct 20, Aleksey Midenkov wrote:
> > > > revision-id: 0cd5377e9ff (mariadb-10.2.31-357-g0cd5377e9ff)
> > > > parent(s): dc716da4571
> > > > author: Aleksey Midenkov 
> > > > committer: Aleksey Midenkov 
> > > > timestamp: 2020-08-17 00:52:35 +0300
> > > > message:
> > > >
> > > > MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal
upon update on versioned partitioned table
> > >
> > > No, I think we need to go deeper here.
> > > The bug is not only not caused by system versioning, it is also not
> > > caused by virtual columns.
> > >
> > > The problem is that partitioning copies record[0] out and then back
> > > to save and restore values. This does not work for blobs, because
> > > record[0] may store the pointer to the memory that Field_blob owns
> > > and can free or reallocate as needed.
> > >
> > > Normally it's not the problem because typically after row reads the
> > > engine owns the memory, not Field_blob. When a virtual column is set
> > > to a calculated value - then Field_blob owns the memory and we have
> > > this bug.
> >
> > Yes, you right... Actually engine owning blobs is not good too because
> > engine does not guarantee to keep previous records blobs.
>
> I think engine owning blobs is safe. Because they belong to different
> handlers, and every handler is pretty much guaranteed to keep the blob
> for the last read row until the next row is read.


Agreed.

>
>
> > My solution is to make a special mem_root (m_ordered_root) and copy
> > the blobs to m_ordered_root when the record is stored into
> > m_ordered_rec_buffer. That m_ordered_root is freed when
> > m_ordered_rec_buffer is destroyed (m_ordered_rec_buffer can be
> > allocated on m_ordered_root too). When the record is restored from
> > m_ordered_rec_buffer the table fields are made to own blobs: we can
> > destroy m_ordered_root any time and the table->record[0] pointers will
> > still be valid.
>
> Just from your description it doesn't look like a good idea,
> MEM_ROOT can be only freed as a whole, so you'll keep allocating memory
> for blobs there and it'll keep growing, remembering all blobs you ever
> had in the table.


Right. I figured that out after I wrote the reply and made an additional
patch with the array of mem_roots.

>
>
> But I'll see the patch, perhaps I misunderstood what you're doing.
>
> If I'm right though, then I'd suggest to look only at the case when
> Field_blob owns the memory. And to avoid copying blobs - they are big,
> moving them back and forth is going to take time. A fast solution could
> be to take the ownership of the blob memory from the Field_blob. And
> then give it back. Using either String::swap or a pair of
> String::release and String::reset - this won't need to allocate or
> memcpy anything.

This complicates things a little and makes the patch a bit less universal:
we assert that only table->field can own and free the blob buffer otherwise
it is safe. But ok, I made one more patch for your consideration:

https://github.com/MariaDB/server/commits/2f9bbf392072361175c889cbf826f11c9880d678

>
>
> For example, every queue element has a `String backup`. It's empty, not
> alloced. no buffer, nothing. When a Field_blob is put into a queue, you
> backup::swap(Field_blob::value). When it's popped you swap again. This
> should work even if the memory is owned by the engine and the
> Field_blob::value is empty. Of course, there may be more than one blob,
> so it should really be an array of backup strings.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



--
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 0cd5377e9ff: MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table

[Aleksey Midenkov]

Hi Sergei!

On Wed, Oct 21, 2020 at 4:19 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Oct 20, Aleksey Midenkov wrote:
> > revision-id: 0cd5377e9ff (mariadb-10.2.31-357-g0cd5377e9ff)
> > parent(s): dc716da4571
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2020-08-17 00:52:35 +0300
> > message:
> >
> > MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon
> update on versioned partitioned table
> >
> > update_virtual_fields() may reallocate Field_blob::value buffer due to
> > different reasons (f.ex. different types, see
> > Field_str::memcpy_field_possible()). But the buffer address may be
> > already in m_ordered_rec_buffer which is then used in ordered index
> > scan queue.
> >
> > The patch updates blob virtual fields when the ordered index scan
> > queue is sorted as well as the top record is restored from the queue.
> >
> > New VCOL_UPDATE_BLOBS mode is introduced to update only blobs. Swap is
> > not needed as it was already done if it was needed.
> >
> > Related to ea1b2504
>
> No, I think we need to go deeper here.
> The bug is not only not caused by system versioning, it is also not
> caused by virtual columns.
>
> The problem is that partitioning copies record[0] out and then back to
> save and restore values. This does not work for blobs, because record[0]
> may store the pointer to the memory that Field_blob owns and can free or
> reallocate as needed.
>
> Normally it's not the problem because typically after row reads the
> engine owns the memory, not Field_blob. When a virtual column is set to
> a calculated value - then Field_blob owns the memory and we have this
> bug.
>

Yes, you right... Actually engine owning blobs is not good too because
engine does not guarantee to keep previous records blobs. My solution is to
make a special mem_root (m_ordered_root) and copy the blobs to
m_ordered_root when the record is stored into m_ordered_rec_buffer. That
m_ordered_root is freed when m_ordered_rec_buffer is destroyed
(m_ordered_rec_buffer can be allocated on m_ordered_root too). When the
record is restored from m_ordered_rec_buffer the table fields are made to
own blobs: we can destroy m_ordered_root any time and the table->record[0]
pointers will still be valid.


>
> But there are engines that use field->store() to set values - in this
> case Field_blob will own the memory too. Here's a test case with
> FederatedX:
>
> =
> source include/have_partition.inc;
> source include/have_sequence.inc;
> source have_federatedx.inc;
> source include/federated.inc;
> connection slave;
> use federated;
> create table t1_1 (x int, b text, key(x));
> create table t1_2 (x int, b text, key(x));
> connection master;
> eval create table t1 (x int, b text, key(x)) engine=federated
>   partition by range columns (x) (
>   partition p1 values less than (40) connection='mysql://root@127.0.0.1:
> $SLAVE_MYPORT/federated/t1_1',
>   partition pn values less than (maxvalue)
> connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2'
> );
> insert t1 select seq, seq from seq_1_to_35;
> insert t1 select seq, repeat(seq, 100) from seq_50_to_90;
> flush tables;
> select x, b from t1 where x > 30 and x < 60 order by x;
> select x, b from t1 where x > 30 and x < 60 order by b;
> drop table t1;
> connection slave;
> drop table t1_1, t1_2;
> source include/federated_cleanup.inc;
> =
>
> first select doesn't trigger ASAN error, but it produces incorrect
> results (I'm showing it here, so that you could create a non-ASAN test
> that could be run on all builders).
>

It turned out that this is the very same MDEV-17573 issue. Multiple
partitions are multiple handlers which interfere with the single
federatedx_io_mysql::current pointer.


>
> the second select gets the same ASAN error as your test below.
>

Slightly modified this test case and added to the patch.


>
> > diff --git a/mysql-test/suite/vcol/r/partition.result
> b/mysql-test/suite/vcol/r/partition.result
> > index bd1353fa145..74c3c3394dc 100644
> > --- a/mysql-test/suite/vcol/r/partition.result
> > +++ b/mysql-test/suite/vcol/r/partition.result
> > @@ -28,3 +28,26 @@ set statement sql_mode= '' for update t1 set i= 1, v=
> 2;
> >  Warnings:
> >  Warning  1906The value specified for generated column 'v' in
> table 't1' has been ignored
> >  drop table t1;
> > +#
> > +# MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal
> upon update on versioned partitioned table

Re: [Maria-developers] 9bf4b92cbc5: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei!


On Wed, Jul 1, 2020 at 12:26 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Jun 29, Aleksey Midenkov wrote:
> > revision-id: 9bf4b92cbc5 (mariadb-10.5.2-168-g9bf4b92cbc5)
> > parent(s): 478301d9b9a
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2020-04-17 17:04:24 +0300
> > message:
> >
> > MDEV-17554 Auto-create new partition for system versioned tables with
history partitioned by INTERVAL/LIMIT
> >
> > == Syntax change ==
> >
> > Keyword AUTO_INCREMENT (or AUTO) enables partition auto-creation.
>
> I still think that AUTO_INCREMENT here looks rather out-of-place,
> I don't know how to explain to users why AUTO_INCREMENT is allowed here.
>
> But if we'll have it, we'll have to support it here for a long time
> for compatibility reasons. "Because somebody might be already using it"
>
> I'd suggest to support only AUTO here.

Sorry, that was just the wrong message. I already removed the
AUTO_INCREMENT keyword.

>
> Or, if the parser can do it easily, a much more readable syntax would be
>
>   create table t1 (x int) with system versioning
>   auto partition by system_time interval 1 hour;
>
> this would be the best, easy to read, matches the documentation (where
> it could be called "auto partitioning"). If it won't cause any difficult
> parser problems, I'd prefer us to use that.

Besides the problems of implementation this probably is not correct
logically. AUTO keyword is applicable only to SYSTEM_TIME partitioning so
it should go after SYSTEM_TIME keyword, but is also applicable only to
rotated partitions so it should go after rotation keywords as well.


>
> > create or replace table t1 (x int) with system versioning
> > partition by system_time interval 1 hour auto_increment;
> >
> > create or replace table t1 (x int) with system versioning
> > partition by system_time limit 1000 auto;
> >
> > Or with explicit partitions:
> >
> > create or replace table t1 (x int) with system versioning
> > partition by system_time interval 1 hour auto
> > (partition p0 history, partition pn current);
> >
> > == Description ==
> >
> > Before executing history-generating DML command add N history
> > partitions, so that N would be sufficient for potentially generated
> > history. N > 1 may be required when history is rotated by INTERVAL and
> > timestamp was jumped to future further than interval value.
> >
> > It is assumed that one DML command will not generate history rows more
> > than specified LIMIT. Otherwise history would overflow generated
> > partition, a warning would be printed, and user action would be
> > required to rebuild partitions.
> >
> > Auto-creation is implemented by synchronous
> > fast_alter_partition_table() call from the thread of the executed DML
> > command before the command itself (by the fallback and retry mechanism
> > similar to Discovery feature, see Open_table_context).
> >
> > Creating history partitions was made by the principle of minimal
> > disruption of the main business process. I.e. when procedure of
> > creation fails it will not (if possible) fail the original DML
> > command. Warning info will display what happened and the last history
> > partition may overflow. In such case partition rebuild is required to
> > correctly display history info. User application may detect warning
> > code ER_VERS_HIST_PART_FAILED and stop execution if that is preferred.
> >
> > The name for newly added partitions are generated like default
> > partition names with extension of MDEV-22155 (which avoids name
> > clashes by extending assignment counter to next free-enough gap).
> >
> > These DML commands trigger auto-creation:
> >
> > * DELETE (including multi-delete, excluding DELETE HISTORY)
> > * UPDATE (including multi-update)
> > * REPLACE (including REPLACE .. SELECT)
>
> INSERT ... ON DUPLICATE KEY UPDATE ?
>

Added.

> > diff --git a/mysql-test/suite/versioning/common.inc
b/mysql-test/suite/versioning/common.inc
> > index 355b571e5a0..b35a5138015 100644
> > --- a/mysql-test/suite/versioning/common.inc
> > +++ b/mysql-test/suite/versioning/common.inc
> > @@ -6,6 +6,7 @@ if (!$TEST_VERSIONING_SO)
> >  source include/have_innodb.inc;
> >
> >  set @@session.time_zone='+00:00';
> > +set @@global.time_zone='+00:00';
>
> Why is that? I'd expect your auto-adding of partitions to happen in the
> context of a session, using session time zone.

This is for the "Concurrent DML" test case. New connections do not get
current session time

Re: [Maria-developers] 7d593466a22: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field

[Aleksey Midenkov]

Hi, Sergei!

I tried variations of second and third options. The
gcol.gcol_keys_innodb depends on the return status and error code
thrown. It turns out that simplest solution is to use
Counting_error_handler alone.

The updated patch is

cd9cab54aac56f0f0171fa661fedffdbb4915edf (bb-10.2-midenok)

On Thu, Apr 23, 2020 at 2:48 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> I don't see what you've changed. We've discussed that fix and that one
> isn't supposed to swap Diagnostics_area's like that. And in your new
> patch you do exactly the same.
>
> Possible correct approaches:
>
> * don't return in_use->is_error(), return the return value of
>   vf->vcol_info->expr->walk() || vf->vcol_info->expr->save_in_field()
>   This means that Item_field::update_vcol_processor() should also
>   do the same, I suspect
>
> * Use thd->push_internal_handler() and Counting_error_handler.
>   Or, better, Turn_errors_to_warnings_handler with counting.
>   This is the simplest one.
>
> there's a third option:
>
> * always return 0, because, looking how it's used, I don't really see
>   how update_virtual_field() can ever get an error. But it's not a
>   particularly future-proof approach. And I just might be wrong about
>   errors.
>
> On Apr 23, Aleksey Midenkov wrote:
> > revision-id: 7d593466a22 (mariadb-10.2.28-4-g7d593466a22)
> > parent(s): 7bc26de591c
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-11-07 10:45:21 +0300
> > message:
> >
> > MDEV-20015 Assertion `!in_use->is_error()' failed in 
> > TABLE::update_virtual_field
> >
> > Preserve and restore statement DA.
> >
> > update_virtual_field() is called as part of index rebuild in
> > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish.
> >
> > Assertion in update_virtual_field() was put as part of MDEV-16222
> > because update_virtual_field() returns in_use->is_error(). The idea:
> > wrongly mixed semantics of error status before update_virtual_field()
> > and the status returned by update_virtual_field(). The former can
> > falsely influence the latter.
> >
> > Preserve global error status and run update_virtual_field() with clear
> > DA since no matter how SQL command is finished it must update the
> > index after bulk INSERT.
> >
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



--
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b97e45651d1: MDEV-16937 Strict SQL with system versioned tables causes issues

[Aleksey Midenkov]

Sergei,

On Tue, May 19, 2020 at 1:52 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On May 19, Aleksey Midenkov wrote:
> > > > diff --git a/mysql-test/suite/versioning/engines.combinations 
> > > > b/mysql-test/suite/versioning/engines.combinations
> > > > index 26b5bab23f1..57e2af6cd06 100644
> > > > --- a/mysql-test/suite/versioning/engines.combinations
> > > > +++ b/mysql-test/suite/versioning/engines.combinations
> > > > @@ -7,5 +7,10 @@ default-storage-engine=innodb
> > > >  [myisam]
> > > >  default-storage-engine=myisam
> > > >
> > > > +[traditional]
> > > > +default-storage-engine=myisam
> > > > +sql-mode=traditional
> > >
> > > I'm not sure about it. It's, of course, safe to run all tests with every
> > > possible settings, but it really blows up testing times.
> > >
> > > May be it'd make sense to have a more targeted test here?
> >
> > Versioning suite without "traditional":
> >
> > real0m32.805s
> > user0m44.173s
> > sys 0m17.346s
> >
> > With "traditional":
> >
> > real0m34.107s
> > user0m50.038s
> > sys 0m20.001s
>
> what value of --parallel did you use? with --mem or without?

With --parallel=4; without --mem. Here are the numbers without both options:

real2m19.709s
user0m42.374s
sys 0m17.671s

real2m21.355s
user0m45.825s
sys 0m18.726s

>
> > IMO not much duration difference. OTOH testing everything with
> > sql-mode=traditional looks useful.
>
> Okay, as you like. It was just a suggestion, our test suite is quite
> slow as it is. But a couple of seconds is fine if they add enough value.
>
> > > >  [heap]
> > > >  default-storage-engine=memory
> > > > +
> > > > diff --git a/mysql-test/suite/versioning/r/select.result 
> > > > b/mysql-test/suite/versioning/r/select.result
> > > > index 3569268ce1d..68df246af6b 100644
> > > > --- a/mysql-test/suite/versioning/r/select.result
> > > > +++ b/mysql-test/suite/versioning/r/select.result
> > > > @@ -45,7 +45,7 @@ ASOF_x  y
> > > >  7107
> > > >  8108
> > > >  9109
> > > > -select x as FROMTO_x, y from t1 for system_time from timestamp '0-0-0 
> > > > 0:0:0' to timestamp @t1;
> > > > +select x as FROMTO_x, y from t1 for system_time from timestamp 
> > > > '1970-01-01 00:00:00' to timestamp @t1;
> > >
> > > Does this test use a fixed time zone?
> >
> > Yes, this test uses fixed time zone +00:00 set in common.inc
>
> Sorry, missed that. I did look :(
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b97e45651d1: MDEV-16937 Strict SQL with system versioned tables causes issues

[Aleksey Midenkov]

 Hi, Sergei!

The patches with your comments applied are:

e8ad2360d82b5652b23360dc7ed78c4a276ea514 (bb-10.3-midenok)
9aea660d7078143a6d3e50106136bd380c188ff0 (bb-10.4-midenok2)

On Sun, May 17, 2020 at 11:59 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> First - is commit b97e45651d1 the one I was supposed to review?

Yes.

>
> On May 17, Aleksey Midenkov wrote:
> > revision-id: b97e45651d1 (mariadb-10.4.7-33-gb97e45651d1)
> > parent(s): 7587975bf06
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-08-19 11:58:07 +0300
> > message:
> >
> > MDEV-16937 Strict SQL with system versioned tables causes issues
> >
> > Vers SQL: respect system fields in NO_ZERO_DATE mode.
> >
> > This is the subject for refactoring in tempesta-tech/mariadb#379
> >
> > diff --git a/mysql-test/suite/versioning/engines.combinations 
> > b/mysql-test/suite/versioning/engines.combinations
> > index 26b5bab23f1..57e2af6cd06 100644
> > --- a/mysql-test/suite/versioning/engines.combinations
> > +++ b/mysql-test/suite/versioning/engines.combinations
> > @@ -7,5 +7,10 @@ default-storage-engine=innodb
> >  [myisam]
> >  default-storage-engine=myisam
> >
> > +[traditional]
> > +default-storage-engine=myisam
> > +sql-mode=traditional
>
> I'm not sure about it. It's, of course, safe to run all tests with every
> possible settings, but it really blows up testing times.
>
> May be it'd make sense to have a more targeted test here?

Versioning suite without "traditional":

real0m32.805s
user0m44.173s
sys 0m17.346s

With "traditional":

real0m34.107s
user0m50.038s
sys 0m20.001s

IMO not much duration difference. OTOH testing everything with
sql-mode=traditional looks useful.

>
> > +
> >  [heap]
> >  default-storage-engine=memory
> > +
> > diff --git a/mysql-test/suite/versioning/r/select.result 
> > b/mysql-test/suite/versioning/r/select.result
> > index 3569268ce1d..68df246af6b 100644
> > --- a/mysql-test/suite/versioning/r/select.result
> > +++ b/mysql-test/suite/versioning/r/select.result
> > @@ -45,7 +45,7 @@ ASOF_x  y
> >  7107
> >  8108
> >  9109
> > -select x as FROMTO_x, y from t1 for system_time from timestamp '0-0-0 
> > 0:0:0' to timestamp @t1;
> > +select x as FROMTO_x, y from t1 for system_time from timestamp '1970-01-01 
> > 00:00:00' to timestamp @t1;
>
> Does this test use a fixed time zone?

Yes, this test uses fixed time zone +00:00 set in common.inc

> If not, is 1970-01-01 00:00:00 a valid timestamp in all time zones?
> If unsure, better use 1971, that will definitely be fine everywhere.

Ok, got that hint.

>
> >  FROMTO_x y
> >  0100
> >  1101
> > diff --git a/sql/field.cc b/sql/field.cc
> > index 969c32a5180..f5580239a9f 100644
> > --- a/sql/field.cc
> > +++ b/sql/field.cc
> > @@ -11139,6 +11139,13 @@ bool Field::save_in_field_default_value(bool 
> > view_error_processing)
> >  {
> >THD *thd= table->in_use;
> >
> > +  /*
> > + TODO: refactor setting the system fields via default_value mechanism.
> > + This condition will go away as well as other conditions with 
> > VERS_SYSTEM_FIELD.
> > +  */
>
> MDEV?
> There's a pull request, but it is marked as obsolete

Updated comment to:
MDEV-19597 Refactor TABLE::vers_update_fields() via stored virtual columns

>
> > +  if (flags & VERS_SYSTEM_FIELD)
> > +return false;
> > +
> >if (unlikely(flags & NO_DEFAULT_VALUE_FLAG &&
> > real_type() != MYSQL_TYPE_ENUM))
> >{
> > diff --git a/sql/unireg.cc b/sql/unireg.cc
> > index d019b5f8a75..484ad0334e4 100644
> > --- a/sql/unireg.cc
> > +++ b/sql/unireg.cc
> > @@ -1060,7 +1060,8 @@ static bool make_empty_rec(THD *thd, uchar *buff, 
> > uint table_options,
> >  !f_bit_as_char(field->pack_flag))
> >null_count+= field->length & 7;
> >
> > -error= make_empty_rec_store_default(thd, regfield, 
> > field->default_value);
> > +if (!(field->flags & VERS_SYSTEM_FIELD))
> > +  error= make_empty_rec_store_default(thd, regfield, 
> > field->default_value);
>
> will the field be initialized at all? It should have something, even
> bzero will do, but we shouldn't dump uninited memory to frm.

Fixed.

>
> >  delete regfield; // Avoid memory leaks
> >  if (error)
> >goto err;
> >
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Review for MDEV-19751.

[Aleksey Midenkov]

Hi, Alexey!

Actually the bug is applicable only to primary key and only when
default field list is set (i.e. empty KEY() clause). So updated the
patch accordingly. Please peek again.

On Wed, May 13, 2020 at 2:40 PM Alexey Botchkov  wrote:
>
> Hello, Aleksey!
>
> > commit 5ef168e00f7b8c821463e46535a4beccb47ec8ea
> +  if (key_info->flags & HA_INVISIBLE_KEY)
> +continue;
>
> I'd add || !(key_info->flags & HA_NOSAME).
> Can the non-unique index participate in the KEY partitioning?
>
> +  for (uint kp= 0; kp < key_info->user_defined_key_parts; ++kp)
> +  {
> +const KEY_PART_INFO _part= key_info->key_part[kp];
> +for (Field **part_field= tab_part_info->part_field_array;
> +*part_field; ++part_field)
> +{
> +  if (*part_field == key_part.field)
> +  {
> +*partition_changed= TRUE;
> +goto search_finished;
> +  }
> +} // for (part_field)
> +  } // for (key_part)
>
> So you check if the key has any of the partitioning fields in it.
> In this case we are going to disallow INPLACE with no reason if the
> unrelated key is removed just containing the primary key column.
> I'd say we should check for the exact match, when the key has
> all the part_field_array in that exact order. Disagree?
>
> And for the test - please add the case for a table with no PRIMARY KEY, bug 
> with the UNIQUE instead.
>
> Best regards!
> HF
>


-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 9941c6a3179: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi, Sergei!

I think there should be hard limit on maximum partitions created, as
there can be manual timestamp jump. F.ex. timestamp jump for 20 years
creates >17 hourly partitions. What the limit should be? I'd
better refuse to create more than 100 partitions.

On Thu, Apr 9, 2020 at 9:25 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 09, Aleksey Midenkov wrote:
> > > > +#
> > > > +# MDEV-17554 Auto-create new partition for system versioned tables 
> > > > with history partitioned by INTERVAL/LIMIT
> > > > +#
> > > > +# Don't auto-create new partition on DELETE HISTORY:
> > > > +create or replace table t (a int) with system versioning
> > > > +partition by system_time limit 1000 auto_increment;
> > >
> > > this looks like a hack, I think we need a dedicated syntax for that.
> > > but I couldn't think of anything good now.
> > >
> > > ok, I see that you allow both AUTO_INCREMENT and AUTO.
> > > May be better just to use AUTO?
> >
> > AUTO_INCREMENT looks more explanatory. Isn't it?
>
> Not to me. In my opinion it looks
> 1) confusing

Not more confusing than say "create table" and "create user". It is
natural feature of verbs to be applied to different nouns.

> 2) hackish, a.k.a. "they didn't want to introduce a new keyword, so
>picked an existing one that looked at least remotely relevant"

I don't think it looks like a hack. Partitions number is
auto-incremented, isn't it? Besides, "auto-increment" is best sounding
feature name: "auto-create" sounds inconvenient, "auto-add" sounds
even more clumsy.

But to avoid long dispute on this rather minor topic I removed
AUTO_INCREMENT keyword.

>
> > To be continued...
>
> I've attached a proof-of-concept patch that adds a partition using
> fallback-and-retry mechanism of open_table().

Thanks! Here is the final part of my reply to your review:


On Tue, Apr 7, 2020 at 3:03 PM Sergei Golubchik  wrote:
>
...
> > +set timestamp= unix_timestamp('2000-01-01 00:00:00');
> > +affected rows: 0
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time interval 1 hour auto_increment (
> > +partition p1 history,
> > +partition p3 history,
> > +partition pn current);
> > +affected rows: 0
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 
> > 00:00:00' AUTO_INCREMENT
> > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE)
> > +affected rows: 1
> > +insert into t1 values (1);
> > +affected rows: 1
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 
> > 00:00:00' AUTO_INCREMENT
> > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE)
> > +affected rows: 1
> > +set timestamp= unix_timestamp('2000-01-01 01:00:00');
> > +affected rows: 0
> > +update t1 set x= x + 3;
> > +affected rows: 1
> > +info: Rows matched: 1  Changed: 1  Inserted: 1  Warnings: 0
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 
> > 00:00:00' AUTO_INCREMENT
> > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE,
> > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE)
> > +affected rows: 1
> > +set timestamp= unix_timestamp('2000-01-01 02:00:00');
> > +affected rows: 0
> > +update t1 set x= x + 4;
> > +affected rows: 1
> > +info: Rows matched: 1  Changed: 1  Inserted: 1  Warnings: 0
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PA

Re: [Maria-developers] 9941c6a3179: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi, Sergei!

On Tue, Apr 7, 2020 at 3:03 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Apr 07, Aleksey Midenkov wrote:
> > revision-id: 9941c6a3179 (mariadb-10.5.2-164-g9941c6a3179)
> > parent(s): 920c3c6b237
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2020-04-06 08:05:43 +0300
> > message:
> >
> > MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
>
> See some comment below, please
>
> > diff --git a/mysql-test/suite/versioning/common.inc 
> > b/mysql-test/suite/versioning/common.inc
> > index 355b571e5a0..b35a5138015 100644
> > --- a/mysql-test/suite/versioning/common.inc
> > +++ b/mysql-test/suite/versioning/common.inc
> > @@ -6,6 +6,7 @@ if (!$TEST_VERSIONING_SO)
> >  source include/have_innodb.inc;
> >
> >  set @@session.time_zone='+00:00';
> > +set @@global.time_zone='+00:00';
>
> Why is that? I understand you might've needed it when a partition was added
> in a separate thread, but why now?

There is concurrency case in partition.test it is needed for other connections.

>
> >  select ifnull(max(transaction_id), 0) into @start_trx_id from 
> > mysql.transaction_registry;
> >  set @test_start=now(6);
> >
> > diff --git a/mysql-test/suite/versioning/r/delete_history.result 
> > b/mysql-test/suite/versioning/r/delete_history.result
> > index cb865a835b3..2e4a2bf9974 100644
> > --- a/mysql-test/suite/versioning/r/delete_history.result
> > +++ b/mysql-test/suite/versioning/r/delete_history.result
> > @@ -154,3 +154,18 @@ select * from t1;
> >  a
> >  1
> >  drop table t1;
> > +#
> > +# MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
> > +#
> > +# Don't auto-create new partition on DELETE HISTORY:
> > +create or replace table t (a int) with system versioning
> > +partition by system_time limit 1000 auto_increment;
>
> this looks like a hack, I think we need a dedicated syntax for that.
> but I couldn't think of anything good now.
>
> ok, I see that you allow both AUTO_INCREMENT and AUTO.
> May be better just to use AUTO?

AUTO_INCREMENT looks more explanatory. Isn't it?

>
> > +delete history from t;
> > +show create table t;
> > +TableCreate Table
> > +tCREATE TABLE `t` (
> > +  `a` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT
> > +PARTITIONS 2
> > +drop table t;
> > diff --git a/mysql-test/suite/versioning/r/partition.result 
> > b/mysql-test/suite/versioning/r/partition.result
> > index a7047cbd11b..660d2c81961 100644
> > --- a/mysql-test/suite/versioning/r/partition.result
> > +++ b/mysql-test/suite/versioning/r/partition.result
> > @@ -289,11 +291,27 @@ x
> >  6
> >  7
> >  8
> > -## rotation by INTERVAL
> > +# Auto-create history partitions
> > +create or replace table t1 (x int) with system versioning
> > +partition by system_time limit 1000 auto_increment;
> > +lock tables t1 write;
> > +insert into t1 values (1);
> > +update t1 set x= x + 1;
> > +unlock tables;
> > +show create table t1;
> > +TableCreate Table
> > +t1   CREATE TABLE `t1` (
> > +  `x` int(11) DEFAULT NULL
> > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING
> > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT
> > +PARTITIONS 3
>
> this test needs a comment, I don't understand what's happening here.
> why lock tables?
>

It's just another case for LOCK TABLES. Moved it to better place.

> > +#
> > +# Rotation by INTERVAL
> > +#
> >  create or replace table t1 (x int)
> >  with system versioning
> >  partition by system_time interval 0 second partitions 3;
> > -ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for 
> > 'INTERVAL'
> > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for 
> > INTERVAL
> >  create table t1 (i int) with system versioning
> >  partition by system_time interval 6 day limit 98;
> >  ERROR 42000: You have an error in your SQL syntax; check the manual that 
> > corresponds to your MariaDB server version for the right syntax to use near 
> > 'limit 98' at line 2
> > @@ -302,7 +320,7 @@ partition by system_time interval 10 year partitions 3;
> >  ERROR 22003: TIMESTAMP value is out of range in 'INTERVAL'
> >  # INTERVAL and ALTER TABLE
> >  create or replace tabl

Re: [Maria-developers] bd98ef106c8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Wed, Feb 19, 2020 at 8:27 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Feb 18, Aleksey Midenkov wrote:
> > On Mon, Feb 17, 2020 at 8:23 PM Sergei Golubchik  wrote:
> > > On Feb 17, Aleksey Midenkov wrote:
> > > > On Sat, Nov 9, 2019 at 12:03 AM Sergei Golubchik  
> > > > wrote:
> > > > >
> > > > > 1.
> > > > > It should be possible to enable/disable auto-creation.  For
> > > > > example, CREATE TABLE ... PARTITION BY SYSTEM_TIME ...
> > > > > PARTITIONS AUTO; this solves few problems at once:
> > > > > * a user explicitly tells when auto-creation should work
> > > >
> > > > Done.
> > > >
> > > > > * you don't need to worry "if this name is already occupied"
> > > >
> > > > I have to. There can be partitions created by hand.
> > >
> > > Here I meant that partitions can be either AUTO or manually created.
> > > So if a user had specified AUTO there can be no manually added
> > > partitions.  Which makes the implementation simpler.
> >
> > I believe current implementation is better: compatibility with old
> > syntax and existing tables require just to add one keyword to make
> > things work. I don't believe one little loop makes it more complex.
> > OTOH forced syntax difference is something users nor programmers don't
> > like.
>
> I'm just saying, the implementation can be made notably simpler if we
> won't allow manually and automatically created partitions in the same
> table. But if you want to support that and solve problems that it
> creates  - okay.
>
> > > > > * you can define that these partitions can never overflow (for
> > > > > INTERVAL)
> > > > > * if you know that AUTO partitions never overflow, you can keep
> > > > > the old behavior for ALTER TABLE ADD PARTITION.
> > > >
> > > > Fast ADD is performance consideration. Making data copy on
> > > > auto-creation is feature killer.
> > >
> > > Agree. I meant that if partitions never overflow, then ADD will
> > > never need copying.
> >
> > Agree. The old behavior will be returned in MDEV-21747.
>
> Great, I'll do a new review when it's done, then.
>
> > > Using the parser in the main thread is easy, we do that in many
> > > places. It would be the last reason for me to do a separate thread.
> >
> > I'm not familiar with that. Can we run SQL command from another SQL
> > command easily?
>
> Not "run SQL command" yet. But "parse SQL command" — yes.
> Just grep for "parse_sql" for examples.

Yes, I'm familiar with such isolated parsing. I'm not familiar with
isolated parsing and running, that's what I meant and what the code
does. It looked simplest way to run in separate thread to me like any
connection does.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] bd98ef106c8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Sergei,

On Mon, Feb 17, 2020 at 8:23 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On Feb 17, Aleksey Midenkov wrote:
> > Hi Sergei!
> >
> > On Sat, Nov 9, 2019 at 12:03 AM Sergei Golubchik  wrote:
> > >
> > > Hi, Aleksey!
> > >
> > > 1.
> > > It should be possible to enable/disable auto-creation.  For example,
> > > CREATE TABLE ... PARTITION BY SYSTEM_TIME ...  PARTITIONS AUTO; this
> > > solves few problems at once:
> > > * a user explicitly tells when auto-creation should work
> >
> > Done.
> >
> > > * you don't need to worry "if this name is already occupied"
> >
> > I have to. There can be partitions created by hand.
>
> Here I meant that partitions can be either AUTO or manually created. So
> if a user had specified AUTO there can be no manually added partitions.
> Which makes the implementation simpler.

I believe current implementation is better: compatibility with old
syntax and existing tables require just to add one keyword to make
things work. I don't believe one little loop makes it more complex.
OTOH forced syntax difference is something users nor programmers don't
like.

>
> > > * you can define that these partitions can never overflow (for
> > > INTERVAL)
> > > * if you know that AUTO partitions never overflow, you can keep the
> > > old behavior for ALTER TABLE ADD PARTITION.
> >
> > Fast ADD is performance consideration. Making data copy on
> > auto-creation is feature killer.
>
> Agree. I meant that if partitions never overflow, then ADD will never
> need copying.

Agree. The old behavior will be returned in MDEV-21747.

>
> > > 2.
> > > Separate thread is an interesting solution. Nicely avoids lots of
> > > problems. Still:
> > > * it's asynchronous, all tests have to take it into account
> > > * it's racy. "low probability" or not, with our number of users they'll
> > >   hit it and rightfully will report it as a bug
> > > * if LOCK TABLE prevents it, partitions can overflow
> > >
> > > But I think these problems are easier to solve than those we'll face
> > > if auto-creation will happen in the main connection thread.
> > >
> > > So, let's keep your approach with a thread.
> > >
> > > But instead of going through the parser and mysql_execute_command,
> > > create a function that takes a TABLE or a TABLE_SHARE. And adds a
> > > partition there. It'll fix the "racy" part. This function can be
> > > called from a new thread.
> > >
> > > As for the LOCK TABLES - if you're under LOCK TABLES, you can simply
> > > call that function (to add a partition) directly at the end of the
> > > main INSERT/UPDATE/DELETE statement. It'll solve the last problem,
> > > LOCK TABLES won't prevent auto-creation.
> >
> > The whole point of thread was to use parser. For direct alter I'd
> > prefer to call it at the end of the main statement. Besides:
>
> I don't understand that. Using the parser in the main thread is easy, we
> do that in many places. It would be the last reason for me to do a
> separate thread.

I'm not familiar with that. Can we run SQL command from another SQL
command easily?

>
> > 1. I'm not sure if it is possible to work under MDL ticket from other
> > thread. There will be assertion failures
> > DBUG_ASSERT(thd->mdl_context.is_lock_owner());
> >
> > 2. The threads must synchronized, so no big difference from
> > single-threaded solution. I don't see how separate thread helps.
>
> Okay. Single-threaded works too.
> The tricky part is that you cannot do DDL (adding a partition is DDL)
> under a DML metadata lock.
>
> But I've just realized that it's solvable.
>
> PARTITION BY LIMIT can overflow, even now, it's always approximate, so
> you can add a new partition in the main thread after the main statement.
> It doesn't matter if the statement has inserted more rows than LIMIT.
>
> For PARTITION BY INTERVAL it doesn't work, it has to be rotated before
> the main statement. Here you can use the existing fallback-and-retry
> approach. Take a lock, check the rotation time. Fallback and retry with
> a stronger MDL, if needed. In fact, the same approach will work for
> PARTITION BY LIMIT too, so there is no need to have two different
> implementations for partition rotation.

Rotation and auto-increment are different things. There is buffer of
ready empty partitions for rotation.

>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] bd98ef106c8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT

[Aleksey Midenkov]

Hi Sergei!

On Sat, Nov 9, 2019 at 12:03 AM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> 1.
> It should be possible to enable/disable auto-creation.
> For example, CREATE TABLE ... PARTITION BY SYSTEM_TIME ... PARTITIONS AUTO;
> this solves few problems at once:
> * a user explicitly tells when auto-creation should work

Done.

> * you don't need to worry "if this name is already occupied"

I have to. There can be partitions created by hand.

> * you can define that these partitions can never overflow (for INTERVAL)
> * if you know that AUTO partitions never overflow, you can keep the old
>   behavior for ALTER TABLE ADD PARTITION.

Fast ADD is performance consideration. Making data copy on
auto-creation is feature killer.

>
> 2.
> Separate thread is an interesting solution. Nicely avoids lots of
> problems. Still:
> * it's asynchronous, all tests have to take it into account
> * it's racy. "low probability" or not, with our number of users they'll
>   hit it and rightfully will report it as a bug
> * if LOCK TABLE prevents it, partitions can overflow
>
> But I think these problems are easier to solve than those we'll face if
> auto-creation will happen in the main connection thread.
>
> So, let's keep your approach with a thread.
>
> But instead of going through the parser and mysql_execute_command,
> create a function that takes a TABLE or a TABLE_SHARE. And adds a
> partition there. It'll fix the "racy" part. This function can be called
> from a new thread.
>
> As for the LOCK TABLES - if you're under LOCK TABLES, you can simply
> call that function (to add a partition) directly at the end of the main
> INSERT/UPDATE/DELETE statement. It'll solve the last problem, LOCK
> TABLES won't prevent auto-creation.

The whole point of thread was to use parser. For direct alter I'd
prefer to call it at the end of the main statement. Besides:

1. I'm not sure if it is possible to work under MDL ticket from other
thread. There will be assertion failures
DBUG_ASSERT(thd->mdl_context.is_lock_owner());

2. The threads must synchronized, so no big difference from
single-threaded solution. I don't see how separate thread helps.

>
> A couple of style comments:
> 1. create a one-liner make_partition_name (or something) with:
>
> sprintf(move_ptr, "p%u", i)
>
> and use it both in create_default_partition_names() and in your new
> code. Just to make sure partition names are always generated uniformly
> by the same function and if we'll want to change it to "pp%u" it should
> be done only in one place.
>

Done.

> 2. don't overload operators, use, say,
>
> bool greater_than(size_t seconds)
>

Done.

> But overall it's pretty good. A nice idea with a separate thread.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>
> On Nov 07, Aleksey Midenkov wrote:
> > revision-id: bd98ef106c8 (mariadb-10.4.7-55-gbd98ef106c8)
> > parent(s): 6d1fa01e8f1
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-08-19 12:01:14 +0300
> > message:
> >
> > MDEV-17554 Auto-create new partition for system versioned tables with 
> > history partitioned by INTERVAL/LIMIT
> >
> > When there are E empty partitions left, auto-create N new empty
> > partitions.
> >
> > This scheme must not allow partition overflow. I.e. E-fill time must
> > not exceed N-creation time. This means that low values for INTERVAL
> > and LIMIT must not be allowed for auto-creation. In case when overflow
> > is detected there is no need to do anything special: a warning will be
> > issued and the user will run manual rebuild to redistribute records
> > correctly. This is important because automatic ADD must be done fast,
> > without forced rebuild, by the obvious reason.
> >
> > Initial version implements hard-coded values of 1 for E and N. As well
> > as auto-creation threshold MinInterval = 1 hour, MinLimit = 1000.
> >
> > The name for newly added partition will be first chosen as "pX", where
> > X is partition number and "p" is hard-coded name prefix. If this name
> > is already occupied, the X will be incremented until the resulting
> > name will be free to use.
> >
> > Auto-creation mechanism is applied to every table having LIMIT or
> > INTERVAL clause. Note that there is no much sense in specifying
> > explicit partition list in this case and this is covered by
> > MDEV-19903. The syntax to explicitly turn it on/off as well as
> > user-defined values for E, N and name prefix is subject for further
> > discussion and feature requests.
> >
> > ALTER TABLE ADD PARTITION is now always fast. If there some history
> > partition overflow occurs manual ALTER TABLE REBUILD PARTITION is
> > needed.



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Hi, Sergei!

On Wed, Jan 1, 2020 at 4:07 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Jan 01, Aleksey Midenkov wrote:
> > On Fri, Dec 27, 2019 at 8:22 PM Sergei Golubchik wrote:
> > > On Dec 22, Aleksey Midenkov wrote:
> >
> > My concern is mostly about our plugins. Let the third-party plugins
> > use that API, but our plugins are not restricted to it?
>
> Unfortunately it doesn't work that way. Third-party plugins use
> everything that our plugins are using, because that's what plugin
> authors are looking at and where they're copying from.
>
> Our plugins should be clean as much as possible, they set an example.
>

Okay, but what is wrong having two APIs: C and C++? Let's start defining
one.


>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Hi, Sergei!

On Fri, Dec 27, 2019 at 8:22 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Dec 22, Aleksey Midenkov wrote:
> > > >
> > > > Particularly to this function I don't like its name, semantics and
> > > > signature.
> > >
> > > As far as the service is concerned, "don't like" is too weak an
> > > argument, changing a service comes with a compatibility cost.
> >
> > Of course, if there is compatibility issue. That's not the case for
> > our in-tree plugins, I guess?
>
> You cannot base your decision on the belief that all plugins are
> in-tree. This API was created open for third-party plugins to use.
> So, we have to assume that third party plugins exist
> (and they do exist, check github, sourceforce, and jira).
>

My concern is mostly about our plugins. Let the third-party plugins use
that API, but our plugins are not restricted to it?


>
> > > > Now, to the THD::make_clex_string() and THD::make_lex_string().
> > > > These methods should not be in THD at all. Its monolithic design
> > > > with million of different methods looks to me as a huge mess
> > > > accumulated across long time.  There was no need to create proxies
> > > > when there would not be such a large class in the first place.
> > >
> > > They could be methods of a MEM_ROOT. One shouldn't need a complete
> > > THD to allocate memory from a memroot.
> >
> > I believe, that's not much better than THD method. THD, MEM_ROOT are
> > generic classes, they should know nothing about classes they provide
> > services for.
>
> okay
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Hi, Sergei!

On Thu, Dec 19, 2019 at 11:32 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Dec 18, Aleksey Midenkov wrote:
> > On Wed, Dec 18, 2019 at 7:23 PM Sergei Golubchik 
> wrote:
> > > On Dec 18, Aleksey Midenkov wrote:
> > > > On Wed, Dec 18, 2019 at 1:11 PM Sergei Golubchik 
> wrote:
> > > > >
> > > > > On Dec 18, Aleksey Midenkov wrote:
> > > > > >
> > > > > > We better go away from this C service layer of thd_*()
> > > > > > functions between server and plugins and use class methods
> > > > > > instead.
> > > > >
> > > > > Why is it better? Isn't it just the syntax sugar?
> > > >
> > > > Having local class interfaces is easier to maintain. Additional
> > > > API layer is development costs overhead.
> > >
> > > That's neglectable, thd_make_lex_string() needs next to no
> > > maintainance.
> >
> > Particularly to this function I don't like its name, semantics and
> > signature.
>
> As far as the service is concerned, "don't like" is too weak an
> argument, changing a service comes with a compatibility cost.
>

Of course, if there is compatibility issue. That's not the case for our
in-tree plugins, I guess?


>
> But for the internal server use we can change thd->make_lex_string(),
> indeed.
>
> > They are over-complicated considering its frequent use. I could
> > justify such function if it was used rarely, but when you have 8 calls
> > of it subsequently -- it is ugly. And it takes time to check the
> > signature because it is not clear what is NULL and what is TRUE. I
> > mean you periodically have to check it and this is multiplied by
> > infinite future, so yes, it takes time.
>
> Yes, it could've been better, I agree. But see above.
>
> > Now, to the THD::make_clex_string() and THD::make_lex_string(). These
> > methods should not be in THD at all. Its monolithic design with
> > million of different methods looks to me as a huge mess accumulated
> > across long time.  There was no need to create proxies when there
> > would not be such a large class in the first place.
>
> They could be methods of a MEM_ROOT. One shouldn't need a complete THD
> to allocate memory from a memroot.
>

I believe, that's not much better than THD method. THD, MEM_ROOT are
generic classes, they should know nothing about classes they provide
services for.


>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Sergei,

On Wed, Dec 18, 2019 at 7:23 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Dec 18, Aleksey Midenkov wrote:
> > Sergei,
> >
> > On Wed, Dec 18, 2019 at 1:11 PM Sergei Golubchik 
> wrote:
> >
> > > Hi, Aleksey!
> > >
> > > On Dec 18, Aleksey Midenkov wrote:
> > > > On Wed, Dec 4, 2019 at 10:10 PM Sergei Golubchik wrote:
> > > >
> > > > But it's better to be from the other side:
> > > >
> > > > bool Lex_cstring::strdup(MEM_ROOT *mem_root, const Lex_cstring )
> > > > {
> > > >   // allocate and deep-copy from src to this
> > > > }
> > > >
> > > > I'd really like to use such utility methods instead of C variants
> like
> > > > thd_make_lex_string().
> > > ...
> > > > We better go away from this C service layer of thd_*() functions
> > > > between server and plugins and use class methods instead.
> > >
> > > Why is it better? Isn't it just the syntax sugar?
> >
> > Having local class interfaces is easier to maintain. Additional API
> > layer is development costs overhead.
>
> That's neglectable, thd_make_lex_string() needs next to no maintainance.


Particularly to this function I don't like its name, semantics and
signature. They are over-complicated considering its frequent use. I could
justify such function if it was used rarely, but when you have 8 calls of
it subsequently -- it is ugly. And it takes time to check the signature
because it is not clear what is NULL and what is TRUE. I mean you
periodically have to check it and this is multiplied by infinite future, so
yes, it takes time.

Now, to the THD::make_clex_string() and THD::make_lex_string(). These
methods should not be in THD at all. Its monolithic design with million of
different methods looks to me as a huge mess accumulated across long time.
There was no need to create proxies when there would not be such a large
class in the first place.


>
> It is good for version compatibility control, but we don't have
> > third-party plugins, do we?
>
> I don't know. There definitely were third-party storage engines,
> third-party fulltext parsers, etc.
>
> In fact, I'm sure there are third-party plugins that I know nothing
> about (besides the fact that they exist).
>
> > And we don't strictly use API to get server services into plugin,
> > AFAIK.
>
> This is automatic, any plugin that uses thd_make_lex_string() uses the
> "thd_alloc" service.
>
> > > The only effect I can think of - pure C plugins won't be able to use
> > > thd_make_lex_string() if you replace it with a C++ method.
> >
> > Do we have such plugins and do we need them to stay pure C?
>
> No, not for thd_make_lex_string(). Of all the plugins I know of, only
> storage engines use it at the moment and they all are C++.
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Sergei,

On Wed, Dec 18, 2019 at 1:11 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Dec 18, Aleksey Midenkov wrote:
> > On Wed, Dec 4, 2019 at 10:10 PM Sergei Golubchik wrote:
> >
> > But it's better to be from the other side:
> >
> > bool Lex_cstring::strdup(MEM_ROOT *mem_root, const Lex_cstring )
> > {
> >   // allocate and deep-copy from src to this
> > }
> >
> > I'd really like to use such utility methods instead of C variants like
> > thd_make_lex_string().
> ...
> > We better go away from this C service layer of thd_*() functions
> > between server and plugins and use class methods instead.
>
> Why is it better? Isn't it just the syntax sugar?
>

Having local class interfaces is easier to maintain. Additional API layer
is development costs overhead. It is good for version compatibility
control, but we don't have third-party plugins, do we? And we don't
strictly use API to get server services into plugin, AFAIK.


>
> The only effect I can think of - pure C plugins won't be able to use
> thd_make_lex_string() if you replace it with a C++ method.
>

Do we have such plugins and do we need them to stay pure C?


>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Hi Sergei!

On Wed, Dec 4, 2019 at 10:10 PM Sergei Golubchik  wrote:

> ...
> > +  Lex_cstring *strdup_root(MEM_ROOT _root)
>
> The way you use it, it looks like you really need a constructor, not
> a strdup.
>

On second thought, it can't be a constructor because it can fail with
allocation error.
But it's better to be from the other side:

bool Lex_cstring::strdup(MEM_ROOT *mem_root, const Lex_cstring )
{
  // allocate and deep-copy from src to this
}

I'd really like to use such utility methods instead of C variants like
thd_make_lex_string(). For plugins we can have strdup() method accepting
THD * and compiled inside server (non-inline):

bool Lex_cstring::strdup(THD *thd, const Lex_cstring )
{
  strdup(thd->mem_root, src);
}

We better go away from this C service layer of thd_*() functions between
server and plugins and use class methods instead. We can't use THD directly
though, because we don't want it compiled in plugins. So we compile
non-inline methods in server that are used in plugin.


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Hi, Sergei!

On Fri, Dec 6, 2019 at 5:34 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Dec 05, Aleksey Midenkov wrote:
> > >
> > > > 3. Invalidate referenced shares on:
> > > >
> > > >   - RENAME TABLE
> > > >   - DROP TABLE
> > > >   - RENAME COLUMN
> > > >   - CREATE TABLE
> > > >   - ADD FOREIGN KEY
> > > >
> > > > When foreign table is created or altered by the above operations
> > > > all referenced shares are closed. This blocks the operation while
> > > > any referenced shares are used (when at least one its TABLE
> > > > instance is locked).
> > >
> > > And this is the main question of this email:
> > > Why do you close referenced tables?
> >
> > Table cannot operate with wrong TABLE_SHARE::referenced_keys, so it
> > must be locked anyway whether referenced_keys are modified or
> > reloaded. The choice was made to reload as the first simplest
> > implementation. And unless we know real performance drawback I'd like
> > to keep it that way.  Modifying TABLE_SHARE::referenced_keys requires
> > several different algorithms for the above operations.
>
> I feel that closing tables is a bit too heavy.
>
> By the way, do you open referenced tables for SELECT? Or does
> table->s->referenced_keys stay NULL?
>

Now it stays NULL, but next implementation with FRM files will see foreign
tables list referencing it and it will need to open their shares to update
its referenced_keys. I don't know if we need such level of independence as
to not open foreign tables when not needed. But yes, that could be a
feature to load foreign tables on demand.


>
> > > Minor comments below.
> > > > diff --git a/sql/handler.h b/sql/handler.h
> > > > index e913af1d15d..10984505f70 100644
> > > > --- a/sql/handler.h
> > > > +++ b/sql/handler.h
> > > > @@ -1030,6 +1031,15 @@ struct TABLE_SHARE;
> > > >  struct HA_CREATE_INFO;
> > > >  struct st_foreign_key_info;
> > > >  typedef struct st_foreign_key_info FOREIGN_KEY_INFO;
> > > > +class Table_ident;
> > > > +class FK_list : public List
> > > > +{
> > > > +public:
> > > > +  /* Get all referenced tables for foreign key fk_name. */
> > > > +  bool get(THD *thd, std::set , LEX_CSTRING
> _name, bool foreign);
> > > > +  /* Get all referenced or foreign tables. */
> > > > +  bool get(THD *thd, std::set , bool foreign);
> > >
> > > Seems unnecessary. Copying a list into a std::set _could_ be
> justified, if
> > > you'd later used it for quick checks "if this table belongs to a set" -
> > > something you cannot quickly do with a List.
> > >
> > > But as far as I can see you copy the List into std::set and then
> iterate
> > > this set. This doesn't make much sense, you can iterate the original
> > > list just fine.
> >
> > std::set does duplicate elimination. Iterating a list would require an
> > algorithm to skip the duplicates.
>
> Duplicates? Can there be duplicate foreign keys?
>

No, but duplicates of Table_ident can be, because multiple foreign keys can
reference same table.


> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c91ec05e01b: MDEV-20865 Store foreign key info in TABLE_SHARE

[Aleksey Midenkov]

Hi Sergei!

On Wed, Dec 4, 2019 at 10:10 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> Just a question for now
> (and a couple of style comments, I didn't look at the logic yet)
>
> On Dec 04, Aleksey Midenkov wrote:
> > revision-id: c91ec05e01b (mariadb-10.4.4-427-gc91ec05e01b)
> > parent(s): e6d653b448d
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-11-26 13:04:07 +0300
> > message:
> >
> > MDEV-20865 Store foreign key info in TABLE_SHARE
> >
> > 1. Access foreign keys via TABLE_SHARE::foreign_keys and
> >   TABLE_SHARE::referenced_keys;
> >
> > 2. Remove handler FK interface:
> >
> >   - get_foreign_key_list()
> >   - get_parent_foreign_key_list()
> >   - referenced_by_foreign_key()
>
> Good, that was the goal
>
> > 3. Invalidate referenced shares on:
> >
> >   - RENAME TABLE
> >   - DROP TABLE
> >   - RENAME COLUMN
> >   - CREATE TABLE
> >   - ADD FOREIGN KEY
> >
> > When foreign table is created or altered by the above operations all
> > referenced shares are closed. This blocks the operation while any
> > referenced shares are used (when at least one its TABLE instance is
> > locked).
>
> And this is the main question of this email:
> Why do you close referenced tables?
>

Table cannot operate with wrong TABLE_SHARE::referenced_keys, so it must be
locked anyway whether referenced_keys are modified or reloaded. The choice
was made to reload as the first simplest implementation. And unless we know
real performance drawback I'd like to keep it that way.
Modifying TABLE_SHARE::referenced_keys requires several different
algorithms for the above operations.


> Minor comments below.
> > diff --git a/sql/handler.h b/sql/handler.h
> > index e913af1d15d..10984505f70 100644
> > --- a/sql/handler.h
> > +++ b/sql/handler.h
> > @@ -1030,6 +1031,15 @@ struct TABLE_SHARE;
> >  struct HA_CREATE_INFO;
> >  struct st_foreign_key_info;
> >  typedef struct st_foreign_key_info FOREIGN_KEY_INFO;
> > +class Table_ident;
> > +class FK_list : public List
> > +{
> > +public:
> > +  /* Get all referenced tables for foreign key fk_name. */
> > +  bool get(THD *thd, std::set , LEX_CSTRING
> _name, bool foreign);
> > +  /* Get all referenced or foreign tables. */
> > +  bool get(THD *thd, std::set , bool foreign);
>
> Seems unnecessary. Copying a list into a std::set _could_ be justified, if
> you'd later used it for quick checks "if this table belongs to a set" -
> something you cannot quickly do with a List.
>
> But as far as I can see you copy the List into std::set and then iterate
> this set. This doesn't make much sense, you can iterate the original
> list just fine.
>

std::set does duplicate elimination. Iterating a list would require an
algorithm to skip the duplicates.


>
> > +};
> >  typedef bool (stat_print_fn)(THD *thd, const char *type, size_t
> type_len,
> >   const char *file, size_t file_len,
> >   const char *status, size_t status_len);
> > diff --git a/sql/lex_string.h b/sql/lex_string.h
> > index a62609c6b60..769f4dcbf5e 100644
> > --- a/sql/lex_string.h
> > +++ b/sql/lex_string.h
> > @@ -41,11 +41,47 @@ class Lex_cstring : public LEX_CSTRING
> >  str= start;
> >  length= end - start;
> >}
> > +  Lex_cstring(const LEX_CSTRING )
> > +  {
> > +str= src.str;
> > +length= src.length;
> > +  }
> >void set(const char *_str, size_t _len)
> >{
> >  str= _str;
> >  length= _len;
> >}
> > +  Lex_cstring *strdup_root(MEM_ROOT _root)
>
> The way you use it, it looks like you really need a constructor, not
> a strdup.
>

Probably, yes.


>
> > +  {
> > +Lex_cstring *dst=
> > +(Lex_cstring *) alloc_root(_root, sizeof(Lex_cstring));
> > +if (!dst)
> > +  return NULL;
> > +if (!str)
> > +{
> > +  dst->str= NULL;
> > +  dst->length= 0;
> > +  return dst;
> > +}
> > +dst->str= (const char *) memdup_root(_root, str, length + 1);
> > +if (!dst->str)
> > +  return NULL;
> > +dst->length= length;
> > +return dst;
> > +  }
> > +  bool operator< (const Lex_cstring& rhs) const
> > +  {
> > +return length < rhs.length || (length == rhs.length && memcmp(str,
> rhs.str, length) < 0);
> > +  }
> > +  bool operator== (const Lex_cstring& rhs) const
> > +

Re: [Maria-developers] d5352b8154d: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field

[Aleksey Midenkov]

Sergei,

On Thu, Oct 31, 2019 at 2:03 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Oct 30, Aleksey Midenkov wrote:
> > On Fri, Oct 25, 2019 at 9:44 PM Sergei Golubchik 
> wrote:
> > > On Oct 25, Aleksey Midenkov wrote:
> > > > revision-id: d5352b8154d (mariadb-10.2.25-54-gd5352b8154d)
> > > > parent(s): 1153950ad0a
> > > > author: Aleksey Midenkov 
> > > > committer: Aleksey Midenkov 
> > > > timestamp: 2019-07-22 15:40:06 +0300
> > > > message:
> > > >
> > > > MDEV-20015 Assertion `!in_use->is_error()' failed in
> TABLE::update_virtual_field
> > > >
> > > > Preserve and restore statement DA.
> > >
> > > This is strange. Diagnostics areas aren't supposed to be temporarily
> > > created on the frame, they aren't arenas.
> >
> > There are already several samples of this pattern in the code. This is
> the
> > main (if not the only) usage of set_stmt_da().
>
> One in sql_prepare.cc, mysql_stmt_get_longdata().
> No idea why, it happened during the merge. The first patch used the
> error handler, which is the supposed way of doing this kind of things.
> Then during the merge it was suddenly replaced with this.
>
> Another one in sql_prepare.cc: Ed_connection::execute_direct(). It's
> dead code, doesn't matter.
>
> Yet another in sql_partition.cc, never mind that, partitioning is a huge
> mess, never take any ideas from it.
>
> And the last one in sql_get_diagnostics.cc, which is the way diagnostic
> areas are supposed to be used. That one is correct.
>
> > > Why TABLE::update_virtual_field() is called at all if there's already
> > > an error?
> >
> > update_virtual_field() is called as part of  REPAIR (MDEV-5800
> > <https://jira.mariadb.org/browse/MDEV-5800>) which is done on bulk
> insert
> > finish. I doubt it should consider error status in this case as no matter
> > how SQL command is finished it must update the index.
> >
> > #0  TABLE::update_virtual_field (this=0x7f85b4068388, vf=0x7f85b4066948)
> at /home/midenok/src/mariadb/10.2/src/sql/table.cc:7685
> > #1  0x01105d2b in compute_vcols (info=0x7f85b406a5d8,
> record=0x7f85b4006948 "\377\001", keynum=1) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:683
> > #2  0x011142bd in sort_get_next_record
> (sort_param=0x7f860c05edd8) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/mi_check.c:3657
> > #3  0x01118a33 in sort_key_read (sort_param=0x7f860c05edd8,
> key=0x7f85b4039808) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/mi_check.c:3121
> > #4  0x011650a9 in find_all_keys (info=0x7f860c05edd8, keys=2,
> sort_keys=0x7f85b40397f8, buffpek=0x7f860c05ea20, maxbuffer=0x7f860c05ea54,
> tempfile=0x7f860c05e898, tempfile_for_exceptions=0x7f860c05e728) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/sort.c:312
> > #5  0x01164b8d in _create_index_by_sort (info=0x7f860c05edd8,
> no_messages=1 '\001', sortbuff_size=134216704) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/sort.c:228
> > #6  0x0111763b in mi_repair_by_sort (param=0x7f85b406e300,
> info=0x7f85b406a5d8, name=0x7f860c05f7c0 "./test/t2", rep_quick=1) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/mi_check.c:2401
> > #7  0x01107097 in ha_myisam::repair (this=0x7f85b4068f20,
> thd=0x7f85b4000cf8, param=..., do_optimize=false) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:1271
> > #8  0x011085d3 in ha_myisam::enable_indexes
> (this=0x7f85b4068f20, mode=2) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:1610
> > #9  0x01108db2 in ha_myisam::end_bulk_insert
> (this=0x7f85b4068f20) at
> /home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:1760
> > #10 0x006d0d23 in handler::ha_end_bulk_insert
> (this=0x7f85b4068f20) at
> /home/midenok/src/mariadb/10.2/src/sql/handler.h:2918
> > #11 0x006cd120 in select_insert::abort_result_set
> (this=0x7f85b4012818) at
> /home/midenok/src/mariadb/10.2/src/sql/sql_insert.cc:3959
> > #12 0x007335a7 in handle_select (thd=0x7f85b4000cf8,
> lex=0x7f85b4004820, result=0x7f85b4012818,
> setup_tables_done_option=1073741824) at
> /home/midenok/src/mariadb/10.2/src/sql/sql_select.cc:383
> > #13 0x006eedad in mysql_execute_command (thd=0x7f85b4000cf8) at
> /home/midenok/src/mariadb/10.2/src/sql/sql_parse.cc:4284
> > #14 0x006e8880 in mysql_parse (thd=0x7f85b4000cf8,
> rawbuf=0x7f85b40117f0 "insert into t2 (pk) select a from t1", length=36,
> parser_s

Re: [Maria-developers] f6269a85ad7: MDEV-17553 Enable setting start datetime for interval partitioned history of system versioned tables

[Aleksey Midenkov]

Hi Sergei,

On Thu, Sep 26, 2019 at 4:15 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Sep 26, Aleksey Midenkov wrote:
> > revision-id: f6269a85ad7 (mariadb-10.4.7-38-gf6269a85ad7)
> > parent(s): a9ca752f1a9
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-08-19 11:58:56 +0300
> > message:
> >
> > MDEV-17553 Enable setting start datetime for interval partitioned
> history of system versioned tables
> >
> > * Interactive STARTS syntax
>
> I wouldn't use the word "interactive" here, better say "Explicit STARTS
> syntax"
>

Fixed.


>
> > * SHOW CREATE
> > * Default STARTS rounding depending on INTERVAL type
>
> This is questionable.
> I don't see why it's much better than the old behavior. It kind of makes
> sense, but that's all.
> If it would've worked that way from the start, it would be fine.
> But now, I'm not sure it offers such important benefits to break the
> compatibility for it.
>

Well, I'd like to take my chance on that as it is more user friendly IMO.
F.ex. if you rotate by DAY it is expected to rotate on 00:00 each day, not
at some strange 13:49:52 or whatever the time was when the command was
issued. The latter just doesn't make sense to me. The improvement doesn't
break compatibility with existing tables as they store STARTS value
explicitly.


>
> > * Warn when STARTS timestamp is further than INTERVAL
> >
> > [Closes tempesta-tech/mariadb#574]
> >
> > === Dependency hints (auto-detected by git-deps) ===
> > 336c0139a89 MDEV-16370 row-based binlog events (updates by primary key)
> can not be applied multiple times to system versioned tables
> ^^^
> remove these dependency hints before pushing, please


Of course I remove them always.


>
>
> diff --git a/mysql-test/suite/versioning/r/partition.result
> b/mysql-test/suite/versioning/r/partition.result
> > index 9e532824414..3606c4e407f 100644
> > --- a/mysql-test/suite/versioning/r/partition.result
> > +++ b/mysql-test/suite/versioning/r/partition.result
> > @@ -1,3 +1,4 @@
> > +call mtr.add_suppression("need more HISTORY partitions");
>
> why these warnings suddenly started to show up in the error log?


Actually, they were shown in the error log before:

2019-10-31 11:51:37 4 [Warning] mysqld: Versioned table `test`.`t1`:
partition `p1` is full, add more HISTORY partitions

2019-10-31 11:51:37 4 [Warning] mysqld: Versioned table `test`.`t1`:
partition `p1` is full, add more HISTORY partitions

But I couldn't find how they were suppressed. Neither in original
commit e851c140f4a4.


>
> >  set system_versioning_alter_history=keep;
> >  # Check conventional partitioning on temporal tables
> >  create or replace table t1 (
> > @@ -266,11 +267,11 @@ x
> >  6
> >  insert into t1 values (7), (8);
> >  Warnings:
> > -Warning  4114Versioned table `test`.`t1`: partition `p1` is
> full, add more HISTORY partitions
> > +Warning  4114Versioned table `test`.`t1`: last HISTORY
> partition (`p1`) is out of LIMIT, need more HISTORY partitions
>
> Hmm, "is out of LIMIT" sounds strange, I liked "is full" more.
> Is it that important to say LIMIT or INTERVAL here?
>

I like this message more because it is clear what clause causes the
warning. OTOH just "full" characteristic is vague and is less helpful of
why it is full. Moreover when it is out of INTERVAL, "full" just sounds
strange.


>
> >  ### warn about full partition
> >  delete from t1;
> >  Warnings:
> > -Warning  4114Versioned table `test`.`t1`: partition `p1` is
> full, add more HISTORY partitions
> > +Warning  4114Versioned table `test`.`t1`: last HISTORY
> partition (`p1`) is out of LIMIT, need more HISTORY partitions
> >  select * from t1 partition (p1) order by x;
> >  x
> >  4
> > diff --git a/mysql-test/suite/versioning/r/partition_rotation.result
> b/mysql-test/suite/versioning/r/partition_rotation.result
> > index 69b30a56bd6..f6db36b117b 100644
> > --- a/mysql-test/suite/versioning/r/partition_rotation.result
> > +++ b/mysql-test/suite/versioning/r/partition_rotation.result
> > @@ -55,4 +57,265 @@ i
> >  explain partitions select * from t1 for system_time all where row_end =
> @ts;
> >  id   select_type table   partitions  typepossible_keys
>  key key_len ref rowsExtra
> >  1SIMPLE  t1  p1_p1sp0,p1_p1sp1   #   NULLNULL
> NULLNULL#   #
> > -drop table t1;
> > +## INTERVAL ... STARTS
> > +create or replace table t1 (i int) with system versioning
> > +partition by syste

Re: [Maria-developers] d5352b8154d: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field

[Aleksey Midenkov]

Hi Sergei,

On Fri, Oct 25, 2019 at 9:44 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Oct 25, Aleksey Midenkov wrote:
> > revision-id: d5352b8154d (mariadb-10.2.25-54-gd5352b8154d)
> > parent(s): 1153950ad0a
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-07-22 15:40:06 +0300
> > message:
> >
> > MDEV-20015 Assertion `!in_use->is_error()' failed in
> TABLE::update_virtual_field
> >
> > Preserve and restore statement DA.
>
> This is strange. Diagnostics areas aren't supposed to be temporarily
> created on the frame, they aren't arenas.
>

There are already several samples of this pattern in the code. This is the
main (if not the only) usage of set_stmt_da().


>
> Why TABLE::update_virtual_field() is called at all if there's already
> an error?
>

update_virtual_field() is called as part of  REPAIR (MDEV-5800
<https://jira.mariadb.org/browse/MDEV-5800>) which is done on bulk insert
finish. I doubt it should consider error status in this case as no matter
how SQL command is finished it must update the index.

#0  TABLE::update_virtual_field (this=0x7f85b4068388,
vf=0x7f85b4066948) at
/home/midenok/src/mariadb/10.2/src/sql/table.cc:7685
#1  0x01105d2b in compute_vcols (info=0x7f85b406a5d8,
record=0x7f85b4006948 "\377\001", keynum=1) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:683
#2  0x011142bd in sort_get_next_record
(sort_param=0x7f860c05edd8) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/mi_check.c:3657
#3  0x01118a33 in sort_key_read (sort_param=0x7f860c05edd8,
key=0x7f85b4039808) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/mi_check.c:3121
#4  0x011650a9 in find_all_keys (info=0x7f860c05edd8, keys=2,
sort_keys=0x7f85b40397f8, buffpek=0x7f860c05ea20,
maxbuffer=0x7f860c05ea54, tempfile=0x7f860c05e898,
tempfile_for_exceptions=0x7f860c05e728) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/sort.c:312
#5  0x01164b8d in _create_index_by_sort (info=0x7f860c05edd8,
no_messages=1 '\001', sortbuff_size=134216704) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/sort.c:228
#6  0x0111763b in mi_repair_by_sort (param=0x7f85b406e300,
info=0x7f85b406a5d8, name=0x7f860c05f7c0 "./test/t2", rep_quick=1) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/mi_check.c:2401
#7  0x01107097 in ha_myisam::repair (this=0x7f85b4068f20,
thd=0x7f85b4000cf8, param=..., do_optimize=false) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:1271
#8  0x011085d3 in ha_myisam::enable_indexes
(this=0x7f85b4068f20, mode=2) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:1610
#9  0x01108db2 in ha_myisam::end_bulk_insert
(this=0x7f85b4068f20) at
/home/midenok/src/mariadb/10.2/src/storage/myisam/ha_myisam.cc:1760
#10 0x006d0d23 in handler::ha_end_bulk_insert
(this=0x7f85b4068f20) at
/home/midenok/src/mariadb/10.2/src/sql/handler.h:2918
#11 0x006cd120 in select_insert::abort_result_set
(this=0x7f85b4012818) at
/home/midenok/src/mariadb/10.2/src/sql/sql_insert.cc:3959
#12 0x007335a7 in handle_select (thd=0x7f85b4000cf8,
lex=0x7f85b4004820, result=0x7f85b4012818,
setup_tables_done_option=1073741824) at
/home/midenok/src/mariadb/10.2/src/sql/sql_select.cc:383
#13 0x006eedad in mysql_execute_command (thd=0x7f85b4000cf8)
at /home/midenok/src/mariadb/10.2/src/sql/sql_parse.cc:4284
#14 0x006e8880 in mysql_parse (thd=0x7f85b4000cf8,
rawbuf=0x7f85b40117f0 "insert into t2 (pk) select a from t1",
length=36, parser_state=0x7f860c0625f0, is_com_multi=false,
is_next_command=false) at
/home/midenok/src/mariadb/10.2/src/sql/sql_parse.cc:7760



>
> > diff --git a/sql/table.cc b/sql/table.cc
> > index f5b5bad99cc..65611d78bde 100644
> > --- a/sql/table.cc
> > +++ b/sql/table.cc
> > @@ -7682,15 +7682,25 @@ int TABLE::update_virtual_fields(handler *h,
> enum_vcol_update_mode update_mode)
> >
> >  int TABLE::update_virtual_field(Field *vf)
> >  {
> > -  DBUG_ASSERT(!in_use->is_error());
> > +  Diagnostics_area *stmt_da= NULL;
> > +  Diagnostics_area tmp_stmt_da(in_use->query_id, false, true);
> > +  bool error;
> >Query_arena backup_arena;
> >DBUG_ENTER("TABLE::update_virtual_field");
> > +  if (unlikely(in_use->is_error()))
> > +  {
> > +stmt_da= in_use->get_stmt_da();
> > +in_use->set_stmt_da(_stmt_da);
> > +  }
> >in_use->set_n_backup_active_arena(expr_arena, _arena);
> >bitmap_clear_all(_set);
> >vf->vcol_info->expr->walk(::update_vcol_processor, 0, _set);
> >vf->vcol_info->expr->save_in_field(vf, 0);
> >in_use->restore

Re: [Maria-developers] 4670c85a48c: MDEV-16222 Assertion `0' failed in row_purge_remove_sec_if_poss_leaf on table with virtual columns and indexes

[Aleksey Midenkov]

Hello Sergei!

Actually I was aware of that, but open_purge_table() looks so
non-generic, so I decided to push that down. Well, I agree, it's "kind
of" generic and it will be safer to move a bit higher. But why to
remove DEBUG_ASSERT? update_virtual_field() relies on is_error() from
inside. If error comes from outside it will return TRUE illegally.
Like it happened with current bug. If there were assert in the first
place it didn't took so long to debug.

On Thu, Jul 4, 2019 at 2:18 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Summary: looks ok, but please move clear_error() to the caller
> (ha_innodb.cc) and remove the assert.
>
> On Jul 04, Aleksey Midenkov wrote:
> > revision-id: 4670c85a48c (mariadb-10.4.5-47-g4670c85a48c)
> > parent(s): 0f55a9eb73b
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-06-27 18:05:25 +0300
> > message:
> >
> > MDEV-16222 Assertion `0' failed in row_purge_remove_sec_if_poss_leaf on 
> > table with virtual columns and indexes
> >
> > Cause
> > Stale thd->m_stmt_da->m_sql_errno which is from different invocation.
> >
> > Fix
> > Reset error state before attempt to open table.
> >
> > diff --git a/mysql-test/suite/innodb/r/purge_secondary_mdev-16222.result 
> > b/mysql-test/suite/innodb/r/purge_secondary_mdev-16222.result
> > new file mode 100644
> > index 000..30e8f9800fb
> > --- /dev/null
> > +++ b/mysql-test/suite/innodb/r/purge_secondary_mdev-16222.result
> > @@ -0,0 +1,34 @@
> ...
> > +select * from t1 into outfile 'load.data';
> > +Warnings:
> > +Warning  1287' INTO ;' is 
> > deprecated and will be removed in a future release. Please use 'SELECT 
> >  INTO  FROM...' instead
>
> See the warning, better not to use the deprecated syntax, unless you
> specifically want to test it,
>
> > +load data infile 'load.data' replace into table t1;
> > +set debug_sync= "now WAIT_FOR latch_released";
> > +set global debug_dbug= "-d,ib_purge_virtual_mdev_16222_1";
> > diff --git a/sql/sql_class.cc b/sql/sql_class.cc
> > index d65b6b8ed8d..cb73b7db2de 100644
> > --- a/sql/sql_class.cc
> > +++ b/sql/sql_class.cc
> > @@ -4762,7 +4762,10 @@ TABLE *open_purge_table(THD *thd, const char *db, 
> > size_t dblen,
> >DBUG_ASSERT(!error || !ot_ctx.can_recover_from_failed_open());
> >
> >if (unlikely(error))
> > +  {
> >  close_thread_tables(thd);
> > +thd->clear_error();
>
> We use thd->clear_error() in many places, but over years there were
> quite a few problems with it, and now I'd prefer to avoid it whenever
> possible.
>
> The thing is, you want to remove a specific error on a specific code
> path, but thd->clear_error() removes all errors, and no matter how you
> got here.
>
> Generally intercepting errors with Internal_error_handler is a much
> safer approach.
>
> In this case, though, thd->clear_error() might be okay, but better move
> it up the stack, do it in the caller when there can be only one way to
> reach this code. Not in a kind-of-generically-looking open table helper.
>
> > +  }
> >
> >DBUG_RETURN(error ? NULL : tl->table);
> >  }
> > diff --git a/sql/table.cc b/sql/table.cc
> > index 9ac167b6adb..93c2c5e88a8 100644
> > --- a/sql/table.cc
> > +++ b/sql/table.cc
> > @@ -8240,6 +8240,7 @@ int TABLE::update_virtual_fields(handler *h, 
> > enum_vcol_update_mode update_mode)
> >
> >  int TABLE::update_virtual_field(Field *vf)
> >  {
> > +  DBUG_ASSERT(!in_use->is_error());
>
> Yes, that's what I mean above. There were cases when
> update_virtual_field() was called when is_error() was true and
> is_error() introduced a bug.
>
> I don't quite remember details, but it was something like, after the
> error has happened somewhere, on the way to returning an error, it was
> calling update_virtual_field() and some code in a function down the
> stack was, like
>
>   some_function_which_returns_void();
>   if (thd->is_error()) // the error inside a function
>   { ... handle it ... }
>
> and the error in this case was not caused by anything in the function,
> but existed from before.
>
> The point is, is_error() and clear_error() are global, and you're
> generally interested in local state. Whether _this block_ caused an
> error.  Clear the error caused by _this function call_. And so on, not
> clear an error that might've happened some time somewhere at the
> undefined point in the past before this code line was reached.
>
> >Query_arena backup_arena;
> >DBUG_ENTER("TABLE::update_virtual_field");
> >in_use->set_n_backup_active_arena(expr_arena, _arena);
>
> Regards,
> Sergei
> VP of MariaDB Server Engineering
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 66f931bcd69: MDEV-18727 improve DML operation of System Versioning

[Aleksey Midenkov]

Hi Sergei!

On Fri, Jun 7, 2019 at 8:45 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> Looks ok, thanks!
> No requests to change anything,
> but still a couple of "why" questions, see below
>
> On Jun 07, Aleksey Midenkov wrote:
> > revision-id: 66f931bcd69 (mariadb-10.3.12-100-g66f931bcd69)
> > parent(s): f4484dfdbf2
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2019-03-26 15:04:52 +0300
> > message:
> >
...
> > diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> > index 6915d4d23ca..28b54e0b3b1 100644
> > --- a/sql/sql_select.cc
> > +++ b/sql/sql_select.cc
> > @@ -779,20 +781,35 @@ int SELECT_LEX::vers_setup_conds(THD *thd, TABLE_LIST 
> > *tables)
> >  }
> >}
> >
> > +  bool is_select= false;
> > +  switch (thd->lex->sql_command)
> > +  {
> > +  case SQLCOM_SELECT:
> > +  case SQLCOM_INSERT_SELECT:
> > +  case SQLCOM_REPLACE_SELECT:
> > +  case SQLCOM_DELETE_MULTI:
> > +  case SQLCOM_UPDATE_MULTI:
> > +is_select= true;
> > +  default:
> > +break;
> > +  }
> > +
> >for (table= tables; table; table= table->next_local)
> >{
> > -if (!table->table || !table->table->versioned())
> > +if (!table->table || table->is_view() || !table->table->versioned())
>
> why?

VIEW conditions are processed inside mysql_derived_prepare(). Btw,
this patch was wrong about VIEW update since there was no appropriate
cases in view.test.
New patch is here:
https://github.com/MariaDB/server/pull/1332/commits/a65299bb284636dfac3530e815871edd4da27dd2

as well as additional test cases. And I attached just a fix for VIEW
update bug in this letter.

>
> >continue;
> >
> >  vers_select_conds_t _conditions= table->vers_conditions;
> >
> >  #ifdef WITH_PARTITION_STORAGE_ENGINE
> > -  /*
> > -if the history is stored in partitions, then partitions
> > -themselves are not versioned
> > -  */
> > -  if (table->partition_names && table->table->part_info->vers_info)
> > +Vers_part_info *vers_info;
> > +if (is_select && table->table->part_info &&
> > +(vers_info= table->table->part_info->vers_info))
> > +{
> > +  if (table->partition_names)
>
> Why is that? You don't seem to use vers_info anywhere.
> Nor I see a reason for two nested if()'s intead of one, as before
> (with an additional `is_select &&` condition)

Looks like an outdated patch. In new patch there is branch:

  else if (!vers_conditions.is_set())
  {

But I removed vers_info variable as it doesn't help anything.

>
> >{
> > +/* If the history is stored in partitions, then partitions
> > +themselves are not versioned. */
> >  if (vers_conditions.is_set())
> >  {
> >my_error(ER_VERS_QUERY_IN_PARTITION, MYF(0), table->alias.str);
>
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok


view_fix.diff.gz
Description: application/gzip
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] FYI: usage of F1-F12 keys in mysql client

[Aleksey Midenkov]

By adding something like this in your .inputrc:

$if MYSQL
"\033OP": "use information_schema; show tables;\n"
"\033OQ": "use test;\n"
"\033OR": "select * from "
"\033OS": "show columns from "
"\033[15~":
"\033[17~":
"\033[18~":
"\033[19~":
"\033[20~":
"\033[21~":
"\033[23~":
"\033[24~":
$endif

you may have F1-F12 keys utilized. Different terminals have different
codes (.inputrc doesn't support capability names unfortunately), so
you'll have to find out your own codes via tack utiity. With strong
will and enough patience it is possible to add Ctrl+, Alt+, Shift+ and
their combinations as well.

This works only with MDEV-17609 patch.

-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification

[Aleksey Midenkov]

Hello, Sergei!


On Fri, May 3, 2019 at 8:43 PM Sergei Golubchik  wrote:
>
> Hi, Aleksey!
>
> On May 03, Aleksey Midenkov wrote:
> > revision-id: ef2519fee4e (versioning-1.0.5-17-gef2519fee4e)
> > parent(s): 56145be2951
> > author: Aleksey Midenkov 
> > committer: Aleksey Midenkov 
> > timestamp: 2018-06-28 13:42:09 +0300
> > message:
> >
> > MDEV-16546 System versioning setting to allow history modification
> >
> > 1. Add server variable system_versioning_modify_history which will
> > allow to set values for row_start, row_end in DML operations.
> >
> > 2. If secure_timestamp is YES or REPLICATION,
> > system_versioning_modify_history does not have effect. If
> > secure_timestamp is SUPER, system_versioning_modify_history requires
> > special privilege (same as for setting current timestamp).
>
> I thought more about this idea. We don't really want to have the history
> editable, do we?

Well, I'm thinking about rollback table data to specific point in
time. That could be a useful feature.

> But it's needed for replication, to keep the master and
> slave identical. That's what secure_timestamp is for.
>
> The idea was that this new variable, system_versioning_modify_history,
> will be just a convenience feature, it will not allow history editing
> any more than one can do without it.
>
> But now I suspect that even with secure_timestamp=NO one cannot truly
> edit history. One can only insert new rows with arbitrary timestamps.
> For example, to insert a row with row_start=1000 and row_end=2000, one
> needs to do (if secure_timestamp=NO):
>
>   set timestamp=1000;
>   insert row;
>   set timestamp=2000;
>   delete row;
>
> But I don't see how one can update or delete a history row with
> secure_timestamp=NO.
>
> Now, with a SUPER privilege and secure_timestamp=NO or SUPER, one can
> use the BINLOG command and truly edit the history arbitrarily, by faking
> row events.

I don't really get it why this is so important: since there is some
limitation by configuration and privilege, we are just fine.
Everything can be changed at filesystem level after all.

>
> The conclusion, I believe, is that system_versioning_modify_history
> should allow INSERTs when secure_timestamp=NO, and it should allow
> UPDATE/DELETE only for a SUPER user when secure_timestamp=NO or SUPER.

I don't see a reason to argue on that. The only thing that is not
clear, why we don't allow INSERTs when secure_timestamp=SUPER?

>
> The second thing I don't like at all, is when a table is created like
>
>   CREATE TABLE t1 (a int) WITH SYSTEM VERSIONING
>
> with row_start/row_end implicit. You don't have it in the test, but
> anyway one should be able to load history into such a table, while the
> table does not have row_start and row_end columns. From the user point
> of view these columns don't exist, they're pseudo-columns, like ROWID.
> They just cannot be insertable-into, conceptually. But a user will want
> to restore the history, right? I don't have a solution for this yet :(
> Any ideas?

We don't have to follow the conception if it doesn't help us. Since we
have physical row_start/row_end, we don't have to pretend they don't
exist. Who will win from that?

>
> See below a couple of minor comments about the patch itself.
>
> ...

These are going to be fixed.

>
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org



-- 
All the best,

Aleksey Midenkov
@midenok

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] set_time() on slave and unversioned -> versioned replication

[Aleksey Midenkov]

Hi Sergei!

On Thu, Apr 4, 2019 at 8:59 PM Sergei Golubchik  wrote:

> Hi, Aleksey!
>
> On Apr 04, Aleksey Midenkov wrote:
> > On Thu, Apr 4, 2019 at 5:08 PM Sergei Golubchik 
> wrote:
> >
> > > > > And I didn't want to force the master to include microseconds in
> > > > > every single event for every single user just in case someone
> > > > > would decide to do unversioned->versioned replication.
> > >
> > But is it really an issue: do you know setups where replication
> > communication is a bottleneck?
>
> It's few percent increase of the binlog size. Not much.
>
> > > And it's a general principle - there will be definitely less than 1% of
> > > users, who will use this. Less than 0.1% too. Most probably, less than
> > > 0.01%. So, the remaining 99.99% should not pay the price for it.
> > >
> > Btw, it would be good to see the stats. We have some feedback plugin
> > that does the job, don't we?
>
> Yes.
>
> > > > > Also, I thought that processing of 100 Query_log_event's in
> > > > > a second is not realistic.
> > > >
> > > > Now it fails just with several events. I guess because
> > > > system_time.sec_part is not reset to 0 initially.
> > >
> > > You said yourself that is is reset:
> > >
> > >   start_time_sec_part= system_time.sec_part= 0;
> > >
> > > Initially it's reset in THD::THD() too:
> > >
> > >   system_time.start.val= system_time.sec= system_time.sec_part= 0;
> > >
> > It is synchronized with hardware clock on each set_start_time().
>
> It must be a bug. Hardware clock shouldn't overwrite
> the counter as it comes from the slave.
>

Yes. And there are more complications: for replication log we can check
thd->slave_thread, because it is replayed, well, in a slave thread. But
executing it from client (which is original MDEV-16370 bug) does not
execute it in a slave thread.


>
> Regards,
> Sergei
> Chief Architect MariaDB
> and secur...@mariadb.org
>


-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] set_time() on slave and unversioned -> versioned replication

[Aleksey Midenkov]

On Thu, Apr 4, 2019 at 5:54 PM Aleksey Midenkov  wrote:

>
> But it can't recover correct statements order anyway. The statements came
> from many master threads to single slave thread in some arbitrary order.
> What is the point in ordering them at slave end?
>
>
This is probably incorrect. The order is predefined, so last event is most
important. We just need to overwrite history in case of conflict.

-- 
All the best,

Aleksey Midenkov
@midenok
___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp