Re: [Maria-developers] 56e2cd20ed3: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table

[Sergei Golubchik]

Hi, Nikita,

See, e.g. handler::is_clustering_key() or handler::ha_rnd_pos_by_record()

On May 28, Nikita Malyavin wrote:
> On Wed, 24 May 2023 at 18:32, Sergei Golubchik  wrote:
> 
> > > +bool handler::is_root_handler() const
> > > +{
> > > +  return this == table->file;
> > > +}
> >
> > better make it inline in handler.h
> >
> > Can't 
> TABLE is incomplete in handler.h.
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 56e2cd20ed3: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table

[Sergei Golubchik]

Hi, Nikita,

As far as I'm concerned, not having an assert there is quite fine too :)

On May 27, Nikita Malyavin wrote:
> 
> I suspect I'm able to move the assertion to a more generic place, like
> is_root_handler, but I'd have to check whether a handler's hton is a
> "certainly leaf" one by comparing it with ha_resolve_by_name outputs
> for "myisam" and "innodb".
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] MTR fails on undefined {cpus} array in My/SysInfo.pm on armhf and armel

[Sergei Golubchik]

Hi, Otto,

On May 26, Otto Kekäläinen wrote:
> 
> Kernel upgrade seems plausible root cause to Perl not getting the cpu
> data in the same way as before.
> 
> Even if the root cause is in the kernel/Perl library, as immediate
> mitigation I will try to test some way to wrap the cpus array check in
> code that gracefully degrades if cpus is not found.

You've missed that graceful degradation is already there, by design.

If you'll simply try to wrap the failing function mtr will fail later
anyway.

I've got an idea, could it be that your /proc/cpuinfo is *empty* ?
If not, could you share it, please?

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] MTR fails on undefined {cpus} array in My/SysInfo.pm on armhf and armel

[Sergei Golubchik]

Hi, Otto,

On May 25, Otto Kekäläinen wrote:
> Hi!
> 
> I noticed that the autopkgtests on Debian on armhf and armel that run
> the mariadb-test-run fail on error:
> 
> starting mysql-test-tun.pl...
> Logging: ./mysql-test-run.pl  --force --testcase-timeout=120
> --suite-timeout=540 --retry=3 --
> ...
> Collecting tests...
> Installing system database...
> Can't use an undefined value as an ARRAY reference at
> lib/My/SysInfo.pm line 166.
> 
> Logs for full context:
> https://ci.debian.net/data/autopkgtest/unstable/armel/m/mariadb/33870767/log.gz
> https://ci.debian.net/data/autopkgtest/unstable/armhf/m/mariadb/33870768/log.gz
> 
> This line 166 in mysql-test/lib/My/SysInfo.pm has:
> 
> # Return the number of cpus found
> sub num_cpus {
>   my ($self)= @_;
>   return int(@{$self->{cpus}}) or
> confess "INTERNAL ERROR: No cpus in list";
> }
> 
> Can somebody with strong Perl skills help me understand
> 1) Where this cpus array is inherited from?

cpus is initialized to be an empty list on the line 119:

118   my $self= bless {
119cpus => (),
120   }, $class;

Then it tries to fill it from /proc/cpuinfo (line 67) and `kstat`
(line 95). If nothing worked it'll create one dummy cpu:

145   push(@{$self->{cpus}},
146  {
147   bogomips => DEFAULT_BOGO_MIPS,
148   model_name => "unknown",
149  });

At no point it sets $self->{cpus} to undef, so I don't understand how
that error of yours is even possible.

> 2) Why might it have now regressed in MariaDB 10.11.3 and why is this
> erroring only on armel/armhf but not other architectures?

It doesn't, this file wasn't touched for ages. Could it be an incredibly
ancient perl there? Just a shot in the dark, I don't think it'll explain
much.

Some distro patches, perhaps?

> 3) What should I do wrap this section in so that if cpus is not set,
> it would just default to '1'?

you shouldn't, it already defaults to 1 "unknown" cpu with 2000 bogomips.

if you'll simply hack num_cpus() to return 1, it'll fail later on
something like $self->{cpus}[0]->{bogomips}

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 56e2cd20ed3: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table

[Sergei Golubchik]

Hi, Nikita,

On May 24, Nikita Malyavin wrote:
> revision-id: 56e2cd20ed3 (mariadb-11.0.1-114-g56e2cd20ed3)
> parent(s): 47f5c7ae7e4
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2023-05-10 01:49:42 +0300
> message:
> 
> MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table
> 
> The row events were applied "twice": once for the ha_partition, and one
> more time for the underlying storage engine.
> 
> There's no such problem in binlog/rpl, because ha_partiton::row_logging
> is normally set to false.
> 
> The fix makes the events replicate only when the handler is a root handler.
> We will try to *guess* this by comparing it to table->file. The same
> approach is used in the MDEV-21540 fix, 231feabd. The assumption is made,
> that the row methods are only called for table->file (and never for a
> cloned handler), hence the assertions are added in ha_innobase and
> ha_myisam to make sure that this is true at least for those engines
> 
> Also closes MDEV-31040, however the test is not included, since we have no
> convenient way to construct a deterministic version.
> 
> diff --git a/sql/handler.cc b/sql/handler.cc
> index 3fcbb5171d6..d8aca5bc80b 100644
> --- a/sql/handler.cc
> +++ b/sql/handler.cc
> @@ -7280,7 +7280,7 @@ int handler::binlog_log_row(const uchar *before_record,
>  log_func, row_logging_has_trans);
>  
>  #ifdef HAVE_REPLICATION
> -  if (unlikely(!error && table->s->online_alter_binlog))
> +  if (unlikely(!error && table->s->online_alter_binlog && is_root_handler()))
>  error= binlog_log_row_online_alter(table, before_record, after_record,
> log_func);
>  #endif // HAVE_REPLICATION
> @@ -7800,7 +7800,7 @@ int handler::ha_write_row(const uchar *buf)
>  DBUG_RETURN(error);
>  
>/*
> -NOTE: this != table->file is true in 3 cases:
> +NOTE: is_root_handler() is false in 3 cases:

may be move this comment closer to is_root_handler() ?

>  
>  1. under copy_partitions() (REORGANIZE PARTITION): that does not
> require long unique check as it does not introduce new rows or new 
> index.
> @@ -7810,7 +7810,7 @@ int handler::ha_write_row(const uchar *buf)
>  3. under ha_mroonga::wrapper_write_row()
>*/
>  
> -  if (table->s->long_unique_table && this == table->file)
> +  if (table->s->long_unique_table && is_root_handler())
>{
>  DBUG_ASSERT(inited == NONE || lookup_handler != this);
>  if ((error= check_duplicate_long_entries(buf)))
> @@ -7862,13 +7862,13 @@ int handler::ha_update_row(const uchar *old_data, 
> const uchar *new_data)
>error= ha_check_overlaps(old_data, new_data);
>  
>/*
> -NOTE: this != table->file is true under partition's ha_update_row():
> +NOTE: is_root_handler() is false under partition's ha_update_row():
>  check_duplicate_long_entries_update() was already done by
>  ha_partition::ha_update_row(), no need to check it again for each single
>  partition. Same applies to ha_mroonga wrapper.
>*/
>  
> -  if (!error && table->s->long_unique_table && this == table->file)
> +  if (!error && table->s->long_unique_table && is_root_handler())
>  error= check_duplicate_long_entries_update(new_data);
>table->status= saved_status;
>  
> @@ -7913,6 +7913,12 @@ int handler::ha_update_row(const uchar *old_data, 
> const uchar *new_data)
>return error;
>  }
>  
> +
> +bool handler::is_root_handler() const
> +{
> +  return this == table->file;
> +}

better make it inline in handler.h

> +
>  /*
>Update first row. Only used by sequence tables
>  */
> diff --git a/sql/handler.h b/sql/handler.h
> index fb0eba83c38..9b903e7ebce 100644
> --- a/sql/handler.h
> +++ b/sql/handler.h
> @@ -3502,6 +3502,10 @@ class handler :public Sql_alloc
>  return extra(HA_EXTRA_NO_KEYREAD);
>}
>  
> +protected:
> +  bool is_root_handler() const;
> +
> +public:
>int check_collation_compatibility();
>int check_long_hash_compatibility() const;
>int ha_check_for_upgrade(HA_CHECK_OPT *check_opt);
> diff --git a/storage/innobase/handler/ha_innodb.cc 
> b/storage/innobase/handler/ha_innodb.cc
> index f035437b1d7..94d9da3486b 100644
> --- a/storage/innobase/handler/ha_innodb.cc
> +++ b/storage/innobase/handler/ha_innodb.cc
> @@ -7750,6 +7750,8 @@ ha_innobase::write_row(
>  
>   DBUG_ENTER("ha_innobase::write_row");
>  
> + DBUG_ASSERT(is_root_handler() || table->file->ht != innodb_hton_ptr);

instead of putting this assert into every handler's implementation
and in many methods, why not to have it in handler ha_ methods or may be
even in the is_root_handler itself? You can compare this->ht with
table->file->ht. For example, like

  DBUG_ASSERT(this == table->file || this->ht != table->file->ht);

> +
>   trx_t*  trx = thd_to_trx(m_user_thd);
>  
>   /* Validation checks before we commence write_row operation. */
> @@ -8479,6 +8481,8 @@ ha_innobase::update_row(
>  
>   

Re: [Maria-developers] c75f774d278: MDEV-31058 ER_KEY_NOT_FOUND upon concurrent CHANGE column autoinc and DML

[Sergei Golubchik]

Hi, Nikita,

On May 24, Nikita Malyavin wrote:
> revision-id: c75f774d278 (mariadb-11.0.1-118-gc75f774d278)
> parent(s): 319207175bd
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2023-05-15 19:43:49 +0300
> message:
> 
> MDEV-31058 ER_KEY_NOT_FOUND upon concurrent CHANGE column autoinc and DML
> 
> When column is changed to autoinc, ALTER TABLE may update zero/NULL values,
> if NO_AUTO_VALUE_ON_ZERO mode is not enabled.
> 
> Forbid this for LOCK=NONE for the unreliable cases.
> The cases are described in online_alter_check_autoinc.

just disable online for any ALTER that adds an autoinc attribute to an
existing column. See MDEV.

> diff --git a/mysql-test/main/alter_table_online.test 
> b/mysql-test/main/alter_table_online.test
> index b9e147c812d..7ebe553aabb 100644
> --- a/mysql-test/main/alter_table_online.test
> +++ b/mysql-test/main/alter_table_online.test
> @@ -135,3 +135,71 @@ alter table t1 add d int, lock=none;
>  set system_versioning_alter_history= default;
>  drop table t1;
>  
> +--echo #
> +--echo # MDEV-31058 ER_KEY_NOT_FOUND upon concurrent CHANGE column autoinc 
> +--echo # and DML
> +--echo #
> +create table t (a serial, b int) engine=innodb;
> +--error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON
> +alter table t drop a, modify b serial, algorithm=copy, lock=none;
> +
> +set statement sql_mode= NO_AUTO_VALUE_ON_ZERO for
> +alter table t drop a, modify b serial, algorithm=copy, lock=none;

I don't think it should work, see my comment in the MDEV

> +--echo # Finally good.
> +alter table t modify d int auto_increment, add key(d), 
> +  algorithm=copy, lock=none;

no, it's not good either, as far as I understand

> +
> +drop table t;

lots of unvisible spaces at line ends, please remove them

> +
> +--echo # MDEV-31172 Server crash or ASAN errors in online_alter_check_autoinc
> +create table t (a int, b int, c char(8), key(a,b,c));
> +alter table t modify c int auto_increment key, algorithm=copy;

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c457f237511: MDEV-30984 Online ALTER table is denied with non-informative error messages

[Sergei Golubchik]

Hi, Nikita,

comments below refer to a combined diff of both commits

On May 23, Nikita Malyavin wrote:
> revision-id: c457f237511 (mariadb-11.0.1-124-gc457f237511)
> parent(s): f4b04ec534d
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2023-05-15 21:07:50 +0300
> message:
> 
> MDEV-30984 Online ALTER table is denied with non-informative error messages

Two main thoughts.

First, users will generally don't do show warnings after an error -
mariadb already puts some useful info into a warning after an error in
some (few) cases, and it's always a surprise when I write in a bug
report "use SHOW WARNINGS". It's not very intuitive. And in your case
it's even unnecessary, because your error message is

  LOCK=NONE is not supported. Reason: %s. Try LOCK=SHARED

and "Reason" part is perfectly suited for an explanation that you
generate, instead of a misleading "COPY algorithm requires a lock"
(because it doesn't).

Second thought, your messages are hard-coded and won't be translated
even if the user will configure the server to use, say, spanish or
Japanese error messages. See how it was solved for "COPY algorithm
requires a lock", ER_ALTER_OPERATION_NOT_SUPPORTED_REASON_COPY. But I
think the message will still be comprehensible if you'll simply print
the offending sql clause as the reason, this won't need any
localization. See below where I've shown what your error messages would
look like if you do that.

If you think that's too cryptic, then, please, use the
ER_ALTER_OPERATION_NOT_SUPPORTED_REASON_COPY approach.

Two more comments at the very end but I'll repeat them here, because
they're easy to overlook: AUTO_INCREMENT instead of AUTOINC, and a test
for DROP SYSTEM VERSIONING.

> diff --git a/mysql-test/main/alter_table_online.result 
> b/mysql-test/main/alter_table_online.result
> index 8b726ccb591..bcf7afea9df 100644
> --- a/mysql-test/main/alter_table_online.result
> +++ b/mysql-test/main/alter_table_online.result
> @@ -4,6 +4,10 @@
>  create table t (a int);
>  alter ignore table t add primary key (a), algorithm=copy, lock=none;
>  ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a 
> lock. Try LOCK=SHARED

ERROR 0A000: LOCK=NONE is not supported. Reason: ALTER IGNORE. Try LOCK=SHARED

> +show warnings;
> +LevelCodeMessage
> +Note 1846ALTER IGNORE TABLE is incompatible with LOCK=NONE, 
> ALGORITHM=COPY
> +Error1846LOCK=NONE is not supported. Reason: COPY algorithm 
> requires a lock. Try LOCK=SHARED
>  drop table t;
>  #
>  # MDEV-28771 Assertion `table->in_use&>flushed' failed after ALTER
> @@ -92,12 +96,20 @@ on update cascade) engine=InnoDB;
>  insert into t2 values (1),(2),(3);
>  alter table t2 add c int, algorithm=copy, lock=none;
>  ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a 
> lock. Try LOCK=SHARED

ERROR 0A000: LOCK=NONE is not supported. Reason: FOREIGN KEY ... ON UPDATE 
CASCADE. Try LOCK=SHARED

> +show warnings;
> +LevelCodeMessage
> +Note 1846Tables with CASCADE/SET NULL foreign keys are incompatible with 
> LOCK=NONE, ALGORITHM=COPY
> +Error1846LOCK=NONE is not supported. Reason: COPY algorithm 
> requires a lock. Try LOCK=SHARED
>  alter table t2 add c int, algorithm=inplace, lock=none;
>  create or replace table t2 (b int, foreign key (b)
>  references t1 (a)
>  on delete set null) engine=InnoDB;
>  alter table t2 add c int, algorithm=copy, lock=none;
>  ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a 
> lock. Try LOCK=SHARED

ERROR 0A000: LOCK=NONE is not supported. Reason: FOREIGN KEY ... ON DELETE SET 
NULL. Try LOCK=SHARED

> +show warnings;
> +LevelCodeMessage
> +Note 1846Tables with CASCADE/SET NULL foreign keys are incompatible with 
> LOCK=NONE, ALGORITHM=COPY
> +Error1846LOCK=NONE is not supported. Reason: COPY algorithm 
> requires a lock. Try LOCK=SHARED
>  alter table t2 add c int, algorithm=inplace, lock=none;
>  create or replace table t2 (b int, foreign key (b)
>  references t1 (a)
> @@ -117,6 +129,10 @@ b int references t1 (b) on update cascade) engine=InnoDB;
>  insert into t2 values (1, 1),(2, 2);
>  alter table t2 add c int, algorithm=copy, lock=none;
>  ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a 
> lock. Try LOCK=SHARED

ERROR 0A000: LOCK=NONE is not supported. Reason: FOREIGN KEY ... ON DELETE SET 
NULL. Try LOCK=SHARED

> +show warnings;
> +LevelCodeMessage
> +Note 1846Tables with CASCADE/SET NULL foreign keys are incompatible with 
> LOCK=NONE, ALGORITHM=COPY
> +Error1846LOCK=NONE is not supported. Reason: COPY algorithm 
> requires a lock. Try LOCK=SHARED
>  alter table t2 add c int, algorithm=copy;
>  alter table t2 add d int, algorithm=inplace;
>  drop table t2, t1;
> @@ -132,6 +148,10 @@ period for system_time (row_start, row_end))
>  engine=innodb with system versioning;
>  alter table t1 add c int, algorithm=copy, 

Re: [Maria-developers] Regression in crossbuilding MariaDB

[Sergei Golubchik]

Hi, Otto,

On May 18, Otto Kekäläinen wrote:
> 
> Now I am working on the final fix, which might even be as simple as
> just deleting both copies of the embedded zlib in MariaDB and MariaDB
> Connector C sources, as embedding source code of external projects is
> bad practice anyway.

Right, it's also future proof, no more zlib issues ever.
Users will get a smaller source download too.

bindled zlib is only needed for our bintar builds, never for debs.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Regression in crossbuilding MariaDB

[Sergei Golubchik]

Hi, Otto,

On May 15, Otto Kekäläinen wrote:
> Hello!
> 
> While working on
> https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/46
> (Import new upstream version 10.11.3 to Debian) I noticed that
> amd64->arm64 crossbuilding no longer works.
> 
> The CI shows that Salsa-CI job `test-crossbuild-arm64` no longer
> passes. Based on Salsa-CI runs I can see that it [still worked in
> upstream commit
> e55e761e](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217220)
> but [failed in upstream commit
> 06b443be](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217664),
> thus some of these commits in between regressed it:

Here're relevant lines from the log:
...
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
...
CMake Error: The following variables are used in this project, but they are set 
to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake 
files:
ZLIB_LIBRARY
...
//Path to a library.
ZLIB_LIBRARY_DEBUG:FILEPATH=ZLIB_LIBRARY_DEBUG-NOTFOUND

//Path to a library.
ZLIB_LIBRARY_RELEASE:FILEPATH=ZLIB_LIBRARY_RELEASE-NOTFOUND
...

note that zlib detection is part of cmake, not something we've changed,
so it's could be caused by a change in your build environment.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] da5a72f32d4: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table

[Sergei Golubchik]

Hi, Nikita,

On May 08, Nikita Malyavin wrote:
> Sorry, the assertion should be
> DBUG_ASSERT(table->file == this || dynamic_cast(table->file)
> == NULL);
> 
> Well, the tests seem passing after adding this assertion (and a similar one
> for myisam):
> https://github.com/MariaDB/server/commit/f221bc46e6981c9e7c5d2806faefb1ed086e7bf8
> 
> Okay, I think I can agree on this approach, but then I want these
> assertions to be committed as well.
> dynamic_cast can be replaced with hton comparison, of your choice.

hton comparison, please, otherwise ok.

> Maybe somebody later will need row changes from a separate handler,
> imagine for example cascade foreign key updates, which would have used
> a lookup handler for the cascade changes. For that case, I'll try to
> gather these table->file comparisons under a single method.
> 
> Alternatively, we could introduce some bool handler::root_handler just
> now, without waiting for a demand, but I'm afraid it could become
> another source for bugs. So better later on demand.
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] da5a72f32d4: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table

[Sergei Golubchik]

Hi, Nikita,

On May 08, Nikita Malyavin wrote:
> On Fri, 5 May 2023 at 20:46, Sergei Golubchik  wrote:
> 
> > >  #ifdef HAVE_REPLICATION
> > > -  if (unlikely(!error && table->s->online_alter_binlog))
> > > +  if (unlikely(!error && table->s->online_alter_binlog &&
> > > +   !table->skip_online_logging))
> >
> > this doesn't need any juggling with skip_online_logging.
> > This problem is already solved for, say, long uniques. You can use
> >
> >this == table->file
> >
> > condition to filter out individual partitions and have your
> > binlog_log_row_online_alter called only for the main ha_partition
> > handler.
> 
> This doesn't look reliable -- won't there be any cases when we make a
> change from another handler, not table->file?

So far, that's how it works - updates come through table->file

> If that never happens, we could add DBUG_ASSERT(this == table->file) in
> ha_innodb's/ha_myisam's ha_*_row methods.

Of course, not. That's the whole point. The server calls
ha_partition::ha_write_row, which calls, in turn,
ha_innodb::ha_write_row. For individual partitions this != table->file,
that's how we make sure that certain code (e.g. long unique checks) are
only run once and not repeated per partition.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 24c653be25a: unpack_row: unpack a correct number of fields

[Sergei Golubchik]

Hi, Nikita,

On May 05, Nikita Malyavin wrote:
> revision-id: 24c653be25a (mariadb-11.0.1-123-g24c653be25a)
> parent(s): c1cbda5a2c8
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2023-05-04 20:49:43 +0300
> message:
> 
> unpack_row: unpack a correct number of fields
> 
> diff --git a/sql/rpl_record.cc b/sql/rpl_record.cc
> index 908d2c39d97..30bd80bdfc7 100644
> --- a/sql/rpl_record.cc
> +++ b/sql/rpl_record.cc
> @@ -228,24 +228,30 @@ int unpack_row(const rpl_group_info *rgi, TABLE *table, 
> uint const colcnt,
>const TABLE *conv_table= rpl_data.conv_table;
>DBUG_PRINT("debug", ("Table data: tabldef: %p, conv_table: %p",
> tabledef, conv_table));
> -
> +  bool is_online_alter= rpl_data.is_online_alter();
>DBUG_ASSERT(rgi);
>  
> -  for (field_ptr= begin_ptr; field_ptr < end_ptr && *field_ptr; ++field_ptr)
> +  for (field_ptr= begin_ptr; field_ptr < end_ptr
> + /* In Online Alter conv_table can be wider than
> +   original table, but we need to unpack it all. 
> */
> + && (*field_ptr || is_online_alter);

I failed to understand this, could you please explain? The comment didn't
help much, unfortunately. The test case didn't either.

> +   ++field_ptr)
>{
> +intptr fidx= field_ptr - begin_ptr;

intptr is a variant of int with sizeof(intptr) == sizeof(void*).
it's needed to cast a pointer to int, do some math, cast back.

here you need my_ptrdiff_t or ptrdiff_t

>  /*
>If there is a conversion table, we pick up the field pointer to
>the conversion table.  If the conversion table or the field
>pointer is NULL, no conversions are necessary.
>   */
> -Field *conv_field= conv_table ? conv_table->field[field_ptr - begin_ptr] 
> : NULL;
> +Field *conv_field= conv_table ? conv_table->field[fidx] : NULL;
>  Field *const f= conv_field ? conv_field : *field_ptr;
> +#ifdef DBUG_TRACE
> +Field *dbg= is_online_alter ? f : *field_ptr;
> +#endif
>  DBUG_PRINT("debug", ("Conversion %srequired for field '%s' (#%ld)",
>   conv_field ? "" : "not ",
> - (*field_ptr)->field_name.str,
> - (long) (field_ptr - begin_ptr)));
> + dbg->field_name.str,
> + (long) ()));

eh? you surely meant `(long) fidx` ?
how did it even compile?

>  DBUG_ASSERT(f != NULL);

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c5ce597f06a: MDEV-31043 ER_KEY_NOT_FOUND upon concurrent ALTER and transaction

[Sergei Golubchik]

Hi, Nikita,

On May 04, Nikita Malyavin wrote:
> See my new commit b804bcbc6

You forgot to address the following:

>  if (do_commit)
>  {
> -  // do not set STMT_END for last event to leave table open in altering 
> thd
> -  error= binlog_flush_pending_rows_event(thd, false, true, binlog, 
> );
> -  if (is_ending_transaction)
> +  /*
> +If the cache wasn't reinited to write, then it remains empty after
> +the last write.
> +  */
> +  if (cache.cache_log.type != READ_CACHE && !error)

this is a confusing new condition. are you trying to avoid locking a
mutex for an empty cache? If yes, you can check my_b_bytes_in_cache(),
that'd be more clear.

>{
>  mysql_mutex_lock(binlog->get_log_lock());
>  error= binlog->write_cache(thd, _log);
>  mysql_mutex_unlock(binlog->get_log_lock());
>}
> -  else
> -cache.store_prev_position();
>  }
> -else if (!is_ending_transaction)
> +else if (!commit) // rollback
> +{
>cache.restore_prev_position();
> +}
> +else

add // trans engine, end of statement

> +{
> +  DBUG_ASSERT(!is_ending_transaction);
> +  cache.store_prev_position();
> +}

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] da5a72f32d4: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table

[Sergei Golubchik]

Hi, Nikita,

On May 05, Nikita Malyavin wrote:
> revision-id: da5a72f32d4 (mariadb-11.0.1-114-gda5a72f32d4)
> parent(s): 0d6584c019c
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2023-05-01 01:15:41 +0300
> message:
> 
> MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table
> 
> The row events were applied "twice": once for the ha_partition, and
> one more time for the underlying storage engine.
> 
> There's no such problem in binlog/rpl, because
> ha_partiton::row_logging is normally set to false.

s/ha_partiton/ha_partition/

> The proposed fix adds table->skip_online_logging field, preventing the
> underlying engines from replicating online events. Only upper layer
> engine should do this.
> 
> table->skip_online_logging is initialized to 0 together with the table
> itself. Normally it is always 0, except for the intermediate
> overriding part, once it is checked in handler::binlog_log_row.
> 
> Note on MDEV-31040 (Server crashes in MYSQL_LOG::is_open upon ALTER on
> partitioned table):
> First the solution was to override table->s->online_alter_binlog, and
> set it to NULL during the partitioning operation, which was
> ridiculously wrong: no-one is supposed to modify that field outside
> ALTER! That field write access is protected by EXCLUSIVE metadata
> lock.

I don't think you need to list your different (non-working) attemps to
fix the bug in the commit comment. Just explain the fix that works, not
the fix that doesn't.

> In comparison, table->skip_online_logging is a connection-local field.
> 
> The test is a non-deterministic one. It wasn't convenient to construct
> a deterministic test here, besides for the case that will never be
> repeated.

Is it still the case? Your test in alter_table_online_debug.test with
debug_sync - is it non-deterministic?

> diff --git a/sql/handler.cc b/sql/handler.cc
> index 3fcbb5171d6..a7e229e8f8d 100644
> --- a/sql/handler.cc
> +++ b/sql/handler.cc
> @@ -7280,7 +7280,8 @@ int handler::binlog_log_row(const uchar *before_record,
>  log_func, row_logging_has_trans);
>  
>  #ifdef HAVE_REPLICATION
> -  if (unlikely(!error && table->s->online_alter_binlog))
> +  if (unlikely(!error && table->s->online_alter_binlog &&
> +   !table->skip_online_logging))

this doesn't need any juggling with skip_online_logging.
This problem is already solved for, say, long uniques. You can use

   this == table->file

condition to filter out individual partitions and have your
binlog_log_row_online_alter called only for the main ha_partition
handler.

>  error= binlog_log_row_online_alter(table, before_record, after_record,
> log_func);
>  #endif // HAVE_REPLICATION

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] c5ce597f06a: MDEV-31043 ER_KEY_NOT_FOUND upon concurrent ALTER and transaction

[Sergei Golubchik]

Hi, Nikita,

On May 03, Nikita Malyavin wrote:
> revision-id: c5ce597f06a (mariadb-11.0.1-115-gc5ce597f06a)
> parent(s): da5a72f32d4
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2023-05-01 01:15:41 +0300
> message:
> 
> MDEV-31043 ER_KEY_NOT_FOUND upon concurrent ALTER and transaction
> 
> Non-transactional engines changes are visible immediately the row operation
> succeeds, so we should apply the changes to the online buffer immediately
> after the statement ends.
> 
> diff --git a/sql/log.cc b/sql/log.cc
> index f30ce3bcbd1..1d98455c2b3 100644
> --- a/sql/log.cc
> +++ b/sql/log.cc
> @@ -7701,23 +7701,35 @@ static int binlog_online_alter_end_trans(THD *thd, 
> bool all, bool commit)
>  auto *binlog= cache.sink_log;
>  DBUG_ASSERT(binlog);
>  
> -bool do_commit= commit || !cache.hton->rollback ||
> -cache.hton->flags & HTON_NO_ROLLBACK; // Aria
> +bool do_commit= (commit && is_ending_transaction)
> +|| cache.hton->flags & HTON_NO_ROLLBACK // Aria
> +|| !cache.hton->rollback;
> +
> +error= binlog_flush_pending_rows_event(thd, false, true, binlog, );

you've lost the useful comment about STMT_END.
and you flush events for rollback too that seems like a waste.

I personally would've kept the old structure of if()'s. Like

 -bool do_commit= commit || !cache.hton->rollback ||
 +bool non_trans=   !cache.hton->rollback ||

and

 - if (do_commit)
 + if (commit || non_trans)

with

  -  if (is_ending_transaction)
  +  if (is_ending_transaction || non_trans)

That would've called binlog flush only where it needs to be.

> +
>  if (do_commit)
>  {
> -  // do not set STMT_END for last event to leave table open in altering 
> thd
> -  error= binlog_flush_pending_rows_event(thd, false, true, binlog, 
> );
> -  if (is_ending_transaction)
> +  /*
> +If the cache wasn't reinited to write, then it remains empty after
> +the last write.
> +  */
> +  if (cache.cache_log.type != READ_CACHE && !error)

this is a confusing new condition. are you trying to avoid locking a
mutex for an empty cache? If yes, you can check my_b_bytes_in_cache(),
that'd be more clear.

>{
>  mysql_mutex_lock(binlog->get_log_lock());
>  error= binlog->write_cache(thd, _log);
>  mysql_mutex_unlock(binlog->get_log_lock());
>}
> -  else
> -cache.store_prev_position();
>  }
> -else if (!is_ending_transaction)
> +else if (!commit) // rollback
> +{
>cache.restore_prev_position();
> +}
> +else

add // trans engine, end of statement

> +{
> +  DBUG_ASSERT(!is_ending_transaction);
> +  cache.store_prev_position();
> +}

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] d67c3f88883: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_in

[Sergei Golubchik]

Hi, Alexander,

A question: if the frm will *always* be created in the canonical,
non-ORACLE mode and always parsed in the same mode, will we even need
these schema-qualified native functions?

okay, the schema-qualified trick will help to avoid creating
REGEX_REPLACE_ORACLE and so on. But is it needed to fix MDEV-27744?

More comments below

On Apr 27, Alexander Barkov wrote:
> revision-id: d67c3f3 (mariadb-10.3.33-61-gd67c3f3)
> parent(s): 7355f7b1f5c
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-04-22 15:35:16 +0400
> message:
> 
> MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in 
> row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in 
> row_upd_sec_index_entry (debug) | Corruption
> 
> The crash happened with an indexed virtual column whose
> value is evaluated using a function that has a different meaning
> in sql_mode='' vs sql_mode=ORACLE:
...
good comment

> diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc
> index 59d2ea60715..8e83ee02782 100644
> --- a/sql/sql_partition.cc
> +++ b/sql/sql_partition.cc
> @@ -2591,11 +2591,9 @@ char *generate_partition_syntax_for_frm(THD *thd, 
> partition_info *part_info,
>  HA_CREATE_INFO *create_info,
>  Alter_info *alter_info)
>  {
> -  sql_mode_t old_mode= thd->variables.sql_mode;
> -  thd->variables.sql_mode &= ~MODE_ANSI_QUOTES;
> +  Sql_mode_save_for_frm_handling sql_mode_save(thd);

right. thanks

>char *res= generate_partition_syntax(thd, part_info, buf_length,
>   true, create_info, alter_info);
> -  thd->variables.sql_mode= old_mode;
>return res;
>  }
>  
> diff --git a/sql/unireg.cc b/sql/unireg.cc
> index efbb5e773b4..3b37944afa5 100644
> --- a/sql/unireg.cc
> +++ b/sql/unireg.cc
> @@ -652,6 +649,7 @@ static bool pack_expression(String *buf, 
> Virtual_column_info *vcol,
>  static bool pack_vcols(String *buf, List _fields,
>   List 
> *check_constraint_list)
>  {
> +  Sql_mode_save_for_frm_handling sql_mode_save(current_thd);

well, it was done above pack_vcols(), because the caller had access to
the thd. Why not to keep it that way?

>List_iterator it(create_fields);
>Create_field *field;
>  
> diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc
> index f41414f8ae9..0f5c7519d07 100644
> --- a/sql/item_cmpfunc.cc
> +++ b/sql/item_cmpfunc.cc
> @@ -3292,9 +3292,24 @@ void Item_func_case_simple::print(String *str, 
> enum_query_type query_type)
>  }
>  
>  
> +Item_func_decode_oracle *
> +Item_func_decode_oracle::create(THD *thd, const LEX_CSTRING ,
> +List *item_list)
> +{
> +  if (unlikely(!item_list || item_list->elements < 3))
> +  {
> +wrong_param_count_error(oracle_schema_ref.name(), name);
> +return NULL;
> +  }
> +  return new (thd->mem_root) Item_func_decode_oracle(thd, *item_list);
> +}

why? items are normally created in the Create_xxx::create* methods,
defined in item_create.cc. Why did you delegate that to static
methods in Item_func_xxx ?

> +
> +
>  void Item_func_decode_oracle::print(String *str, enum_query_type query_type)
>  {
> -  str->append(func_name());
> +  print_sql_mode_dependent_name(str, query_type,
> +oracle_schema_ref,
> +Item_func_decode_oracle::func_name());
>str->append('(');
>args[0]->print(str, query_type);
>for (uint i= 1, count= when_count() ; i <= count; i++)
> diff --git a/sql/mysqld.h b/sql/mysqld.h
> index d40e1d170d0..54069c3cd5a 100644
> --- a/sql/mysqld.h
> +++ b/sql/mysqld.h
> @@ -741,12 +741,25 @@ enum enum_query_type
>  QT_ITEM_SUBSELECT_ID_ONLY,
>  
>QT_SHOW_SELECT_NUMBER= (1<<10),
> +
> +  //QT_ITEM_IDENT_DISABLE_DB_TABLE_NAMES= (1 <<11), -- this is taken in 10.5
> +
> +  /// Print sql_mode-dependent functions with the schema qualifier
> +  /// even if the currently implied (by sql_mode) schema is equal to
> +  /// to the function schema, e.g. mariadb_schema.concat('a').
> +  QT_ITEM_FUNC_FORCE_SCHEMA_NAME= (1 << 12),
> +
> +  /// Print for FRM file. Focus on parse-back.
> +  /// e.g. VIEW expressions and virtual column expressions
> +  QT_FOR_FRM= (1 << 13),
> +
>/// This is used for EXPLAIN EXTENDED extra warnings / Be more detailed
>/// Be more detailed than QT_EXPLAIN.
>/// Perhaps we should eventually include 
> QT_ITEM_IDENT_SKIP_CURRENT_DATABASE
>/// here, as it would give better readable results
>QT_EXPLAIN_EXTENDED=  QT_TO_SYSTEM_CHARSET|
> -QT_SHOW_SELECT_NUMBER,
> +QT_SHOW_SELECT_NUMBER|
> +QT_ITEM_FUNC_FORCE_SCHEMA_NAME,

don't do this. let's try to keep the behavior *exactly as before*
for people who don't switch between sql_modes all the time

on the second thought. why do you need 

Re: [Maria-developers] 940d028521f: MDEV-30164 System variable for default collations

[Sergei Golubchik]

Hi, Alexander,

On Mar 10, Alexander Barkov wrote:
> On 3/7/23 2:20 PM, Sergei Golubchik wrote:
> > I'm sorry it took a while.
> >
> > I'm still thinking about the whole thing, it's a rather big change
> > for a really fringe functionality. But I failed to come up with
> > something better so far.
> >
> > Code-wise the patch is mostly fine. See few small comments below,
> > and one slightly larger comment re. replication.
>
> As we discussed on slack, my plan was not to change the hard-coded
> default collations in the collation library.
> So in the library it will always be utf8mb4_general_ci as a default
> collation for utf8mb4. And the server default collations will be
> handled by this new variable character_set_collations.
> I did not mean this variable to be something fringe or temporary.

I see. Then collations set as default this way must be undistinguishable
in behavior from compiled-in defaults.
In particular, SHOW COLLATIONS, I_S.COLLATIONS, etc (if anything) should
show them as default.

And performance considerations become more important, I wasn't paying
much attention to that, as I thought it's a fringe feature.

Like, a linear search in Charset_collation_map_st could be replaced with
a binary. A HASH might be too much overhead, but a binary search in the
array could be worth it. As it's a search that every single user will be
using in the future, perhaps even in every single query. Or even many
times per query.

> > On Mar 07, Alexander Barkov wrote:
> >> revision-id: 940d028521f (mariadb-10.11.1-4-g940d028521f)
> >> parent(s): 71dedd0ebcd
> >> author: Alexander Barkov
> >> committer: Alexander Barkov
> >> timestamp: 2022-12-14 20:00:22 +0400
> >> message:
> >>
> >> MDEV-30164 System variable for default collations
> >>
> >> This patch adds a way to override default collations
> >> (or "character set collations") for desired character sets.
> >>
> >> diff --git a/sql/lex_charset.h b/sql/lex_charset.h
> >> --- a/sql/lex_charset.h
> >> +++ b/sql/lex_charset.h
>
> >> +case 
> >> Lex_exact_charset_extended_collation_attrs_st::TYPE_CHARACTER_SET_COLLATE_EXACT:
> >> +case 
> >> Lex_exact_charset_extended_collation_attrs_st::TYPE_COLLATE_CONTEXTUALLY_TYPED:
> >
> > COLLATE DEFAULT is TYPE_COLLATE_CONTEXTUALLY_TYPED.
> > shouldn't it use the map too?
>
> This method charset_info() is used only in one place, in this method of
> class Column_definition:
...

Thanks for the explanation!

> >> +case 
> >> Lex_exact_charset_extended_collation_attrs_st::TYPE_COLLATE_EXACT:
> >> +  break;
> >> +}
> >> +return m_ci;
> >> +  }
> >> Type type() const
> >> {
> >>   return m_type;
> >> diff --git a/sql/log_event.h b/sql/log_event.h
> >> --- a/sql/log_event.h
> >> +++ b/sql/log_event.h
> >> @@ -2147,6 +2153,8 @@ class Query_log_event: public Log_event
> >> bool sql_mode_inited;
> >> bool charset_inited;
> >>
> >> +  LEX_CSTRING character_set_collations_str;
> >
> > better LEX_CUSTRING, don't you think? It's not even a _str.
>
> Right, it's not _str. It's a binary data. Removed the suffix.
>
> What are benefits of LEX_CUSTRING?

LEX_CSTRING is a string, char*.

LEX_CUSTRING is an array of bytes, uchar*. I think it's more appropriate
here.

> >> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> >> --- a/sql/log_event_server.cc
> >> +++ b/sql/log_event_server.cc
> >> @@ -1194,6 +1194,14 @@ bool Query_log_event::write()
> >>   int2store(start+2, auto_increment_offset);
> >>   start+= 4;
> >> }
> >> +
> >> +  if (thd && thd->variables.character_set_collations.count())
> >> +  {
> >> +*start++= Q_COLLATIONS_SESSION;
> >> +size_t len= 
> >> thd->variables.character_set_collations.to_binary((char*)start);
> >> +start+= len;
> >> +  }
> >
> > Perhaps, detect if it's needed? A cheap way of doing it would be to
> > extend your Elem_st with a query_id. And every time you
> > find_elem_by_charset, you set this elem's query_id to thd->query_id.
> > And here you write only elements with the current query id. If any.
> >
> > Another approach would be to have a bitmap, like
> >
> >uchar used_default_coll_mapping;
> >
> > and in find_elem_by_charset() you set the bit, like
> >
> >used_default_coll_mapping |= 1 << i;
> >
> > and t

Re: [Maria-developers] 8c1ad2a9fe9: MDEV-30633 DATETIME to TIMESTAMP conversion to return maximum timestamp on overflow

[Sergei Golubchik]

Hi, Alexander,

please add tests with fractional seconds.

remember, that in UTC the max timestamp(2) is 2038-01-19 03:14:07.99
and max timestamp(6) is 2038-01-19 03:14:07.99

On Mar 08, Alexander Barkov wrote:
> revision-id: 8c1ad2a9fe9 (mariadb-10.11.1-48-g8c1ad2a9fe9)
> parent(s): ce4a289f1c3
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2023-02-13 17:25:18 +0400
> message:
> 
> MDEV-30633 DATETIME to TIMESTAMP conversion to return maximum timestamp on 
> overflow

a bit more verbose description would be nice

> diff --git a/mysql-test/main/events_bugs.result 
> b/mysql-test/main/events_bugs.result
> --- a/mysql-test/main/events_bugs.result
> +++ b/mysql-test/main/events_bugs.result
> @@ -35,12 +35,34 @@ SET NAMES latin1;
>  set @a=3;
>  CREATE PROCEDURE p_16 () CREATE EVENT e_16 ON SCHEDULE EVERY @a SECOND DO 
> SET @a=5;
>  ERROR HY000: Recursion of EVENT DDL statements is forbidden when body is 
> present
> +SET time_zone='+00:00';
> +SET timestamp=UNIX_TIMESTAMP('2023-02-13 00:00:00');
>  create event e_55 on schedule at 010100 do drop table t;
> -ERROR HY000: Incorrect AT value: '010100'

Hmm, why did you not keep it an error?

> +Warnings:
> +Warning  1292Truncated incorrect timestamp value: '-01-01 
> 00:00:00'
> +Warning  1105Event scheduler is switched off, use SET GLOBAL 
> event_scheduler=ON to enable it.
> diff --git a/mysql-test/main/func_time.result 
> b/mysql-test/main/func_time.result
> --- a/mysql-test/main/func_time.result
> +++ b/mysql-test/main/func_time.result
> @@ -599,19 +599,25 @@ Warnings:
>  Warning  1292Truncated incorrect unixtime value: '2147483648'
>  select unix_timestamp('2039-01-20 01:00:00');
>  unix_timestamp('2039-01-20 01:00:00')
> -NULL
> +2147483647
> +Warnings:
> +Warning  1292Truncated incorrect timestamp value: '2039-01-20 
> 01:00:00'

it's consistent with

  MariaDB [test]> select cast(1e30 as int);
  +-+
  | cast(1e30 as int)   |
  +-+
  | 9223372036854775807 |
  +-+
  1 row in set, 1 warning (0.000 sec)

  Note (Code 1916): Got overflow when converting '1e30' to SIGNED BIGINT. Value 
truncated

good.
Except that it's Note vs Warning. Want to unify this somehow?

>  select unix_timestamp('1968-01-20 01:00:00');
>  unix_timestamp('1968-01-20 01:00:00')
>  NULL

isn't it strange? That overflow is capped to max value, while underflow
is NULL?

> diff --git a/mysql-test/main/select.result b/mysql-test/main/select.result
> --- a/mysql-test/main/select.result
> +++ b/mysql-test/main/select.result
> @@ -3766,11 +3766,15 @@ AND t1.ts BETWEEN "2006-01-01" AND "2006-12-31";
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1SIMPLE  t2  const   PRIMARY PRIMARY 4   const   1   
>  1SIMPLE  t1  range   ts  ts  4   NULL2   Using 
> index condition; Using where
> +Warnings:
> +Warning  1292Truncated incorrect timestamp value: '2999-12-31 
> 00:00:00'

Looks questionable. The query compares timestamp with a datetime. I
think in this case timestamp should be casted to a datetime (so, no
warning), not vice versa.

>  SELECT * FROM t1 LEFT JOIN t2 ON (t1.a=t2.a) WHERE t1.a=30
>  AND t1.ts BETWEEN t2.dt1 AND t2.dt2
>  AND t1.ts BETWEEN "2006-01-01" AND "2006-12-31";
>  ats  a   dt1 dt2
>  30   2006-01-03 23:00:00 30  2006-01-01 00:00:00 2999-12-31 
> 00:00:00
> +Warnings:
> +Warning  1292Truncated incorrect timestamp value: '2999-12-31 
> 00:00:00'
>  DROP TABLE t1,t2;
>  create table t1 (a bigint unsigned);
>  insert into t1 values
> diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc
> --- a/sql/item_timefunc.cc
> +++ b/sql/item_timefunc.cc
> @@ -2793,11 +2793,17 @@ bool Item_func_convert_tz::get_date(THD *thd, 
> MYSQL_TIME *ltime,
>  return true;
>  
>{
> -uint not_used;
> -my_time_tmp= from_tz->TIME_to_gmt_sec(ltime, _used);
> +uint error_code;
> +my_time_tmp= from_tz->TIME_to_gmt_sec(ltime, _code);
>  ulong sec_part= ltime->second_part;
> -/* my_time_tmp is guaranteed to be in the allowed range */
> -if (my_time_tmp)
> +/*
> +  my_time_tmp is guaranteed to be in the allowed range.
> +  Don't perform the conversion in case the source DATETIME was above
> +  TIMESTAMP_MAX_VALUE (and was truncated to TIMESTAMP_MAX_VALUE).

why not?

> +*/   
> +if (my_time_tmp &&
> +(my_time_tmp != TIMESTAMP_MAX_VALUE ||
> + error_code != ER_WARN_DATA_OUT_OF_RANGE))

why `my_time_tmp != TIMESTAMP_MAX_VALUE` ? shouldn't error_code check
be sufficient?

>to_tz->gmt_sec_to_TIME(ltime, my_time_tmp);
>  /* we rely on the fact that no timezone conversion can change sec_part */
>  ltime->second_part= sec_part;

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

Re: [Maria-developers] 940d028521f: MDEV-30164 System variable for default collations

[Sergei Golubchik]

Hi, Alexander,

I'm sorry it took a while.

I'm still thinking about the whole thing, it's a rather big change for
a really fringe functionality. But I failed to come up with something
better so far.

Code-wise the patch is mostly fine. See few small comments below, and
one slightly larger comment re. replication.

On Mar 07, Alexander Barkov wrote:
> revision-id: 940d028521f (mariadb-10.11.1-4-g940d028521f)
> parent(s): 71dedd0ebcd
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-12-14 20:00:22 +0400
> message:
> 
> MDEV-30164 System variable for default collations
> 
> This patch adds a way to override default collations
> (or "character set collations") for desired character sets.
>
> diff --git a/sql/lex_charset.h b/sql/lex_charset.h
> --- a/sql/lex_charset.h
> +++ b/sql/lex_charset.h
> @@ -544,6 +699,20 @@ struct Lex_exact_charset_extended_collation_attrs_st
>{
>  return m_ci;
>}
> +  CHARSET_INFO *charset_info(const Charset_collation_map_st ) const
> +  {
> +switch (m_type)
> +{
> +case Lex_exact_charset_extended_collation_attrs_st::TYPE_CHARACTER_SET:
> +  return map.get_collation_for_charset(m_ci);
> +case TYPE_EMPTY:

Lex_exact_charset_extended_collation_attrs_st::TYPE_EMPTY
(or all case labels without a struct name)

> +case 
> Lex_exact_charset_extended_collation_attrs_st::TYPE_CHARACTER_SET_COLLATE_EXACT:
> +case 
> Lex_exact_charset_extended_collation_attrs_st::TYPE_COLLATE_CONTEXTUALLY_TYPED:

COLLATE DEFAULT is TYPE_COLLATE_CONTEXTUALLY_TYPED.
shouldn't it use the map too?

> +case Lex_exact_charset_extended_collation_attrs_st::TYPE_COLLATE_EXACT:
> +  break;
> +}
> +return m_ci;
> +  }
>Type type() const
>{
>  return m_type;
> diff --git a/sql/log_event.h b/sql/log_event.h
> --- a/sql/log_event.h
> +++ b/sql/log_event.h
> @@ -2147,6 +2153,8 @@ class Query_log_event: public Log_event
>bool sql_mode_inited;
>bool charset_inited;
>  
> +  LEX_CSTRING character_set_collations_str;

better LEX_CUSTRING, don't you think? It's not even a _str.

> +
>uint32 flags2;
>sql_mode_t sql_mode;
>ulong auto_increment_increment, auto_increment_offset;
> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> --- a/sql/log_event_server.cc
> +++ b/sql/log_event_server.cc
> @@ -1194,6 +1194,14 @@ bool Query_log_event::write()
>  int2store(start+2, auto_increment_offset);
>  start+= 4;
>}
> +
> +  if (thd && thd->variables.character_set_collations.count())
> +  {
> +*start++= Q_COLLATIONS_SESSION;
> +size_t len= 
> thd->variables.character_set_collations.to_binary((char*)start);
> +start+= len;
> +  }

Perhaps, detect if it's needed? A cheap way of doing it would be to extend
your Elem_st with a query_id. And every time you find_elem_by_charset,
you set this elem's query_id to thd->query_id. And here you write only
elements with the current query id. If any.

Another approach would be to have a bitmap, like

  uchar used_default_coll_mapping;

and in find_elem_by_charset() you set the bit, like

  used_default_coll_mapping |= 1 << i;

and then, again, print affected collations, if any. Most often
used_default_coll_mapping will likely be zero

one more question. In, say, 10.10->11.1 replication
master and slave will have different default collations, but
thd->variables.character_set_collations will not reflect that.
How do you plan to solve it?

> +
>if (charset_inited)
>{
>  *start++= Q_CHARSET_CODE;
 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6e28a576913: move alloca() definition from all *.h files to one place

[Sergei Golubchik]

Hi, Julius,

this is the combined diff of both commits:

   git diff e1ce867e5ab0c^^ e1ce867e5ab0c

On Feb 10, Julius Goryavsky wrote:
> diff --git a/include/CMakeLists.txt b/include/CMakeLists.txt
> index a7b98a11050..7d87c4a42b3 100644
> --- a/include/CMakeLists.txt
> +++ b/include/CMakeLists.txt
> @@ -104,6 +105,7 @@ ENDMACRO()
>  
>  INSTALL_COMPAT_HEADER(my_global.h "")
>  INSTALL_COMPAT_HEADER(my_config.h "")
> +INSTALL_COMPAT_HEADER(my_alloca.h "")

I wouldn't use INSTALL_COMPAT_HEADER for my_alloca.h

>  INSTALL_COMPAT_HEADER(my_sys.h "")
>  INSTALL_COMPAT_HEADER(mysql_version.h "
>  #include 
> diff --git a/include/my_alloca.h b/include/my_alloca.h
> new file mode 100644
> index 000..fec0320195e
> --- /dev/null
> +++ b/include/my_alloca.h
> @@ -0,0 +1,37 @@
> +/* Copyright (c) 2023, MariaDB Corporation.
> +
> +   This program is free software; you can redistribute it and/or modify
> +   it under the terms of the GNU General Public License as published by
> +   the Free Software Foundation; version 2 of the License.
> +
> +   This program is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +   GNU General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program; if not, write to the Free Software
> +   Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1335 
>  USA */
> +
> +#ifndef MY_ALLOCA_INCLUDED
> +#define MY_ALLOCA_INCLUDED
> +
> +#ifdef _WIN32
> +#include  /*for alloca*/
> +#ifndef alloca

When can alloca be defined?

> +#define alloca _alloca
> +#endif
> +#else
> +#include 

why stdlib.h?

> +#ifdef HAVE_ALLOCA_H
> +#include 
> +#endif
> +#endif
> +
> +#if defined(HAVE_ALLOCA)
> +#if defined(__GNUC__) && !defined(HAVE_ALLOCA_H) && !defined(alloca)

add a comment describing your MinGW use case when this #if is needed

> +#define alloca __builtin_alloca
> +#endif /* GNUC */
> +#endif
> +
> +#endif /* MY_ALLOCA_INCLUDED */
> diff --git a/include/mysql/service_encryption.h 
> b/include/mysql/service_encryption.h
> index 69d205a27e8..cab1418e1d6 100644
> --- a/include/mysql/service_encryption.h
> +++ b/include/mysql/service_encryption.h
> @@ -24,21 +24,19 @@
>*provider* (encryption plugin).
>  */
>  
> +#ifndef MYSQL_ABI_CHECK
> +#include 
> +#endif
> +
>  #ifdef __cplusplus
>  extern "C" {
>  #endif
>  
>  #ifndef MYSQL_ABI_CHECK

why not to put #include  here?

>  #ifdef _WIN32
> -#include 
>  #ifndef __cplusplus
>  #define inline __inline
>  #endif
> -#else
> -#include 

I suspect stdlib.h belongs here, not in my_alloca.h

> -#ifdef HAVE_ALLOCA_H
> -#include 
> -#endif
>  #endif
>  #endif

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ceb75e9bd9e: MDEV-30203: Move mysql symlinks to different package (fixes)

[Sergei Golubchik]

Hi, Daniel,

On Feb 09, Daniel Black wrote:
> revision-id: ceb75e9bd9e (mariadb-10.11.1-152-gceb75e9bd9e)
> parent(s): 526fce28b90
> author: Daniel Black
> committer: Daniel Black
> timestamp: 2023-02-09 12:20:57 +1100
> message:
> 
> MDEV-30203: Move mysql symlinks to different package (fixes)
> 
> Fixes MDEV-30571 MariaDB-client does not pull MariaDB-compat anymore, 
> conflicts with RHEL packages
> 
> Fixes MDEV-30574 MariaDB-server conflicts with MariaDB-common from previous 
> versions
> 
> MariaDB-Server now depends on MariaDB-common 11.0+.
> 
> MariaDB-{Client,Server} append to the Requirements rather than overriding 
> them.
> 
> MariaDB-Server package now depends on the MariaDB-client 11.0+.
> 
> Because of moves between packages MariaDB-Client conflicts with
> MariaDB-server < 11.0.0.
> 
> diff --git a/cmake/cpack_rpm.cmake b/cmake/cpack_rpm.cmake
> index 638f6b2dabb..e88a5493eac 100644
> --- a/cmake/cpack_rpm.cmake
> +++ b/cmake/cpack_rpm.cmake
> @@ -189,7 +189,7 @@ FOREACH(SYM_COMPONENT Server Client)
>SET(CPACK_RPM_${SYM}_PACKAGE_ARCHITECTURE "noarch")
>SET(CPACK_RPM_${SYM}_USER_FILELIST ${ignored})
>STRING(TOLOWER ${SYM_COMPONENT} SYM_COMPONENT_LOWER)
> -  SET(CPACK_RPM_${SYM}_PACKAGE_REQUIRES "MariaDB-${SYM_COMPONENT_LOWER} >= 
> 11.0.0")
> +  SETA(CPACK_RPM_${SYM}_PACKAGE_REQUIRES "MariaDB-${SYM_COMPONENT_LOWER} >= 
> 11.0.0")

why?

>SETA(CPACK_RPM_${SYM_COMPONENT_LOWER}_PACKAGE_RECOMMENDS "MariaDB-${SYM}")
>  ENDFOREACH()
>  
> @@ -205,10 +205,10 @@ SETA(CPACK_RPM_client_PACKAGE_PROVIDES
>"mysql-client"
>"mytop")
>  SETA(CPACK_RPM_client_PACKAGE_CONFLICTS
> -  "MariaDB-server < 10.6.0")
> +  "MariaDB-server < 11.0.0")

because of resolve-stack-dump, ok

>  SETA(CPACK_RPM_common_PACKAGE_CONFLICTS
> -  "MariaDB-server < 10.6.1")
> +  "MariaDB-server < 11.0.0")

why?

>  SETA(CPACK_RPM_devel_PACKAGE_OBSOLETES
>"MySQL-devel")

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] dcd69cbd19a: MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func

[Sergei Golubchik]

Hi, Yuchen,

Thanks!

There are two kinds of comments below, you'll see which is which.
I realize that this patch mainly just moves parts of spider code around.

Some of my comments apply to the new code that didn't exist before and
appeared only in this patch. It's what needs to be fixed (or discussed,
if you disagree) before pushing.

Other comments apply to the old spider code that this patch moved. If
you'd like you can say "I'm just moving spider code around" and ignore
all comments of that kind.

/Sergei

On Jan 22, Yuchen Pei wrote:

> commit dcd69cbd19a
> Author: Yuchen Pei 
> Date:   Tue Jan 3 16:24:04 2023 +1100
> 
> MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func
> 
> Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285)

Because 3836098c29ef will not be, apparently, pushed anywhere,
you'd better put a self-containing comment here. May be use the comment
of 3836098c29ef?

Note that you shouldn't push your 10.4-11.0 versions of the patch directly,
only push the 10.3 one (when ok) and let it propagate to 11.0 through
merges. Your 10.4-11.0 patches are very useful, though, they will serve as
guidelines for whoever will merge this change up.

> to current versions 10.3+ to fix a problem (MDEV-29447) where field
> items that are arguments of a func item may be used before created /
> initialised.
> 
> Signed-off-by: Yuchen Pei 
> 
> diff --git a/storage/spider/spd_db_mysql.cc b/storage/spider/spd_db_mysql.cc
> index e942d1d9063..a886a4e6e99 100644
> --- a/storage/spider/spd_db_mysql.cc
> +++ b/storage/spider/spd_db_mysql.cc
> @@ -4035,6 +4042,170 @@ int spider_db_mbase_util::open_item_func(
>uint alias_length,
>bool use_fields,
>spider_fields *fields
> +) {
> +  DBUG_ENTER("spider_db_mbase_util::check_item_func");

incorrect function name in DBUG_ENTER

> +
> +  int error = check_item_func(item_func, spider, alias,
> +alias_length, use_fields, fields);
> +  if (error)
> +DBUG_RETURN(error);
> +  if (!str)
> +DBUG_RETURN(0);
> +
> +  DBUG_RETURN(print_item_func(item_func, spider, str, alias,
> +alias_length, use_fields, fields));
> +}
> +
> +namespace {
> +  bool item_func_is_timestampdiff(

We generally use static in these cases.
Why would an anonymous namespace be better?

> +const char *func_name,
> +int func_name_length
> +  ) {
> +return func_name_length == 13 &&
> +  !strncasecmp("timestampdiff", func_name, func_name_length);

We build with rtti nowadays, so you can use dynamic_cast instead.
and you won't need func_name and func_name_length.

Ideally you won't need a check for timestampdiff at all

> +  }
> +
> +  bool not_func_should_be_skipped(
> +Item_func *item_func
> +  ){
> +DBUG_ENTER("not_func_should_be_skipped");

add DBUG_ASSERT(item_func->functype() == Item_func::NOT_FUNC);

> +Item **item_list = item_func->arguments();
> +
> +if (item_list[0]->type() == Item::COND_ITEM)
> +{
> +  DBUG_PRINT("info",("spider item_list[0] is COND_ITEM"));
> +  Item_cond *item_cond = (Item_cond *) item_list[0];
> +  if (item_cond->functype() == Item_func::COND_AND_FUNC)
> +  {
> +DBUG_PRINT("info",("spider item_cond is COND_AND_FUNC"));
> +List_iterator_fast lif(*(item_cond->argument_list()));
> +bool has_expr_cache_item = FALSE;
> +bool has_isnotnull_func = FALSE;
> +bool has_other_item = FALSE;
> +while(Item *item = lif++)
> +{
> +  if (
> +item->type() == Item::EXPR_CACHE_ITEM
> +  ) {
> +DBUG_PRINT("info",("spider EXPR_CACHE_ITEM"));
> +has_expr_cache_item = TRUE;
> +  } else
> +  if (
> +item->type() == Item::FUNC_ITEM &&
> +((Item_func *) item)->functype() == Item_func::ISNOTNULL_FUNC
> +  ) {
> +DBUG_PRINT("info",("spider ISNOTNULL_FUNC"));
> +has_isnotnull_func = TRUE;
> +  } else {
> +DBUG_PRINT("info",("spider has other item"));
> +DBUG_PRINT("info",("spider COND type=%d", item->type()));
> +has_other_item = TRUE;
> +  }
> +}
> +if (has_expr_cache_item && has_isnotnull_func && !has_other_item)

So, "not func should be skipped" if it's

   NOT (expr1 AND expr2 AND ...)

and all expr's are either IS NOT NULL or a cached subquery. Why?
IS NULL, IS NOT FALSE, non-cached subqueries, everything else is fine,
IS NOT NULL if it is followed by `AND 2>1` is fine, OR is fine.
Doesn't make much sense to me.


> +{
> +  DBUG_PRINT("info",("spider NOT EXISTS skip"));
> +  DBUG_RETURN(TRUE);
> +}
> +  }
> +}
> +DBUG_RETURN(FALSE);
> +  }
> +}
> +
> +/**
> +  Check if the given item_func and its arguments can be pushed down to
> +  a data node. This function is recursive because we need to also check
> +  the arguments of the item_func.
> +
> +  @return 0: success, otherwise: 

Re: [Maria-developers] 06ce67c644b: MDEV-27653 long uniques don't work with unicode collations

[Sergei Golubchik]

Hi, Alexander,

On Jan 15, Alexander Barkov wrote:
> revision-id: 06ce67c644b (mariadb-10.4.27-41-g06ce67c644b)
> parent(s): 6cb84346e1b
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2023-01-10 18:27:16 +0400
> message:
> 
> MDEV-27653 long uniques don't work with unicode collations
> 
> diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc
> index 213d77f8237..5dd19c877af 100644
> --- a/sql/sql_admin.cc
> +++ b/sql/sql_admin.cc
> @@ -772,11 +772,14 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* 
> tables,
>int check_for_upgrade= file->ha_check_for_upgrade(check_opt);
>  
>if (check_old_types == HA_ADMIN_NEEDS_ALTER ||
> -  check_for_upgrade == HA_ADMIN_NEEDS_ALTER)
> +  check_old_types == HA_ADMIN_NEEDS_UPGRADE ||
> +  check_for_upgrade == HA_ADMIN_NEEDS_ALTER ||
> +  check_for_upgrade == HA_ADMIN_NEEDS_UPGRADE)

eh. So old code was returning HA_ADMIN_NEEDS_ALTER actually quite
intentionally.

REPAIR TABLE was automatically switching to ALTER if these checks were
returning HA_ADMIN_NEEDS_ALTER.

So it didn't matter if the message confusingly said "Please use REPAIR",
because it would've been ALTER internally anyway.

Meaning, I suspect, that the old code was fine and you didn't need to
change `case HA_ADMIN_NEEDS_ALTER` and didn't need to replace
`return HA_ADMIN_NEEDS_UPGRADE` with HA_ADMIN_NEEDS_ALTER.

All you needed to do was to return HA_ADMIN_NEEDS_ALTER from the
check_long_hash_compatibility() method and the rest would've likely
worked automatically. Or at least with much smaller changes.

>{
>  /* We use extra_open_options to be able to open crashed tables */
>  thd->open_options|= extra_open_options;
> -result_code= admin_recreate_table(thd, table);
> +result_code= admin_recreate_table(thd, table) ? HA_ADMIN_FAILED :
> +HA_ADMIN_OK;

good catch

>  thd->open_options&= ~extra_open_options;
>  goto send_result;
>}

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 0bef50e50b5: MDEV-20501: Assertion `maybe_null || !null_value' failed in Item_func_round::date_op

[Sergei Golubchik]

Hi, Sergei,

On Jan 13, Sergei Petrunia wrote:
> revision-id: 0bef50e50b5 (mariadb-10.4.27-33-g0bef50e50b5)
> parent(s): 5db970fc760
> author: Sergei Petrunia
> committer: Sergei Petrunia
> timestamp: 2023-01-04 16:50:12 +0300
> message:
> 
> MDEV-20501: Assertion `maybe_null || !null_value' failed in 
> Item_func_round::date_op
> 
> When the optimizer finds a constant (or system) table $TBL which is empty
> or has no matching row, it would set table->null_row=true. This is done
> even for tables with table->maybe_null==0.
> 
> Then, it would proceed to perform query optimization phases (what for?)
> which will attempt to evaluate Item expressions referring to $TBL.
> 
> Eventually some Item expression will get the value of $TBL.not_null_field,
> get SQL NULL and fail an assertion.
> 
> Fixed by not performing any query optimization steps after we've got
> constant/empty tables with no matching rows.
> 
> Test result changes contain a lot of changes like
> - ... Impossible WHERE noticed after reading const tables
> + ... no matching row in const table
> 
> as well as other changes caused by slightly-different processing of
> the special case of empty constant tables.
> 
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 6a441c5047b..57ffd58f8b4 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -2382,6 +2382,9 @@ int JOIN::optimize_stage2()
>if (subq_exit_fl)
>  goto setup_subq_exit;
>  
> +  if (zero_result_cause)
> +goto setup_subq_exit;

In many other cases it's, like

zero_result_cause= "No matching min/max row";
subq_exit_fl= true;

that is, subq_exit_fl is set whenever zero_result_cause is set.
Meaning, perhaps you should do the same in your new code below
and then you wouldn't need those lines you've added above?

>if (unlikely(thd->check_killed()))
>  DBUG_RETURN(1);
>  
> @@ -5392,7 +5398,32 @@ make_join_statistics(JOIN *join, List 
> _list,
>}
>  }
>} while (ref_changed);
> - 
> +
> +  /*
> +ConstRowNotFoundShortcut-1:
> +Some constant/system tables have mo matching rows. This means that
> +the join operation output will be empty.
> +Short-cut further optimization steps.
> +Note that some query plan steps will still be performed to handle
> +implicit grouping, join result setup, etc.
> +See also: ConstRowNotFoundShortcut-{2,3}.
> +
> +Do not do this if we're optimizing for some UNION's fake_select_lex.
> +We might be running UglySubqueryReoptimization (grep for name) and
> +in this case constant tables are not reliable.
> +  */
> +  if ((join->const_table_map & ~found_const_table_map) &&
> +   !(join->select_lex->master_unit() &&
> + join->select_lex->master_unit()->fake_select_lex == 
> join->select_lex))

why wouldn't you check for `describe` here? "reoptimization" hack is
only needed for EXPLAIN.

> +  {
> +join->zero_result_cause= "no matching row in const table";
> +join->table_count=0;
> +join->const_tables= 0;
> +join->join_tab= NULL;
> +
> +DBUG_RETURN(0);
> +  }
> +
>join->sort_by_table= get_sort_by_table(join->order, join->group_list,
>   join->select_lex->leaf_tables,
>   join->const_table_map);

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 5dcde8f6523: MDEV-27653 long uniques don't work with unicode collations

[Sergei Golubchik]

Hi, Alexander,

Looks good. A couple of comments, see below

On Jan 06, Alexander Barkov wrote:
> revision-id: 5dcde8f6523 (mariadb-10.4.26-64-g5dcde8f6523)
> parent(s): ce443c85547
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-10-28 15:37:44 +0400
> message:
> 
> MDEV-27653 long uniques don't work with unicode collations
>
> diff --git a/mysql-test/main/ctype_utf8.test b/mysql-test/main/ctype_utf8.test
> index cc61c2ae0fe..e2d4e4ab906 100644
> --- a/mysql-test/main/ctype_utf8.test
> +++ b/mysql-test/main/ctype_utf8.test
> @@ -2310,3 +2310,133 @@ VALUES (_latin1 0xDF) UNION VALUES(_utf8'a' COLLATE 
> utf8_bin);
>  --echo #
>  --echo # End of 10.3 tests
>  --echo #
> +
> +
> +--echo #
> +--echo # Start of 10.4 tests
> +--echo #
> +
> +--echo #
> +--echo # MDEV-27653 long uniques don't work with unicode collations
> +--echo #
> +
> +SET NAMES utf8mb3;
> +
> +# CHAR
> +
> +CREATE TABLE t1 (
> +   a CHAR(30) COLLATE utf8mb3_general_ci,
> +   UNIQUE KEY(a) USING HASH
> +);
> +SHOW CREATE TABLE t1;
> +INSERT INTO t1 VALUES ('a');
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +SELECT * FROM t1;
> +DROP TABLE t1;
> +
> +CREATE TABLE t1 (
> +   a CHAR(30) COLLATE utf8mb3_general_ci,
> +   UNIQUE KEY(a(10)) USING HASH
> +);
> +SHOW CREATE TABLE t1;
> +INSERT INTO t1 VALUES ('a');
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +SELECT * FROM t1;
> +DROP TABLE t1;
> +
> +
> +# VARCHAR
> +
> +CREATE TABLE t1 (
> +   a VARCHAR(30) COLLATE utf8mb3_general_ci,
> +   UNIQUE KEY(a) USING HASH
> +);
> +SHOW CREATE TABLE t1;
> +INSERT INTO t1 VALUES ('a');
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +SELECT * FROM t1;
> +DROP TABLE t1;
> +
> +CREATE TABLE t1 (
> +   a VARCHAR(30) COLLATE utf8mb3_general_ci,
> +   UNIQUE KEY(a(10)) USING HASH
> +);
> +SHOW CREATE TABLE t1;
> +INSERT INTO t1 VALUES ('a');
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +SELECT * FROM t1;
> +DROP TABLE t1;
> +
> +
> +# TEXT
> +
> +CREATE TABLE t1 (a TEXT COLLATE utf8mb3_general_ci UNIQUE);
> +SHOW CREATE TABLE t1;
> +INSERT INTO t1 VALUES ('a');
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +SELECT * FROM t1;
> +DROP TABLE t1;
> +
> +CREATE TABLE t1 (
> +  a LONGTEXT COLLATE utf8mb3_general_ci,
> +  UNIQUE KEY(a(10)) USING HASH
> +);
> +SHOW CREATE TABLE t1;
> +INSERT INTO t1 VALUES ('a');
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +SELECT * FROM t1;
> +DROP TABLE t1;
> +
> +
> +# Testing upgrade:
> +# Prior to MDEV-27653, the UNIQUE HASH function errorneously
> +# took into account string octet length.
> +# Old tables should still open and work, but with a wrong results.
> +
> +copy_file std_data/mysql_upgrade/mdev27653_100422_text.frm 
> $MYSQLD_DATADIR/test/t1.frm;
> +copy_file std_data/mysql_upgrade/mdev27653_100422_text.MYD 
> $MYSQLD_DATADIR/test/t1.MYD;
> +copy_file std_data/mysql_upgrade/mdev27653_100422_text.MYI 
> $MYSQLD_DATADIR/test/t1.MYI;
> +SHOW CREATE TABLE t1;
> +SELECT a, OCTET_LENGTH(a) FROM t1 ORDER BY BINARY a;
> +CHECK TABLE t1;
> +
> +# There is already a one byte value 'a' in the table
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('A');
> +
> +# There is already a two-byte value 'ä' in the table
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('Ä');
> +
> +# There were no three-byte values in the table so far.
> +# The below value violates UNIQUE, but it gets inserted.
> +# This is wrong but expected for a pre-MDEV-27653 table.
> +INSERT INTO t1 VALUES ('Ấ');
> +SELECT a, OCTET_LENGTH(a) FROM t1 ORDER BY BINARY a;
> +CHECK TABLE t1;
> +
> +# ALTER FORCE fails: it tries to rebuild the table
> +# with a correct UNIQUE HASH function, but there are duplicates!
> +--error ER_DUP_ENTRY
> +ALTER TABLE t1 FORCE;

please, try ALTER IGNORE TABLE too

> +
> +# Let's remove all duplicate values, so only the one-byte 'a' stays.
> +# ALTER..FORCE should work after that.
> +DELETE FROM t1 WHERE OCTET_LENGTH(a)>1;
> +ALTER TABLE t1 FORCE;
> +
> +# Make sure that 'a' and 'ä' cannot co-exists any more,
> +# because the table was recreated with a correct UNIQUE HASH function.
> +--error ER_DUP_ENTRY
> +INSERT INTO t1 VALUES ('ä');
> +DROP TABLE t1;
> +
> +--echo #
> +--echo # End of 10.4 tests
> +--echo #
> diff --git a/sql/table.cc b/sql/table.cc
> index b9260853381..97f71284005 100644
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -1267,7 +1286,11 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, 
> TABLE *table,
>list_item= new (mem_root) Item_field(thd, keypart->field);
>  field_list->push_back(list_item, mem_root);
>}
> -  Item_func_hash *hash_item= new(mem_root)Item_func_hash(thd, 
> *field_list);
> +
> +  Item_func_hash *hash_item= make_unique_hash_func(thd, mem_root,
> +   
> table->s->mysql_version,
> +   field_list);

would be good to fix CHECK ... FOR UPGRADE too.

> +
>

Re: [Maria-developers] 92ff948d021: MDEV-29231 View returns wrong value with SQL_MODE 'NO_BACKSLASH_ESCAPES'

[Sergei Golubchik]

Hi, Oleksandr,

On Jan 06, Oleksandr Byelkin wrote:
> revision-id: 92ff948d021 (mariadb-10.3.37-53-g92ff948d021)
> parent(s): 180b2bcd538
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-12-05 20:42:03 +0100
> message:
> 
> MDEV-29231 View returns wrong value with SQL_MODE 'NO_BACKSLASH_ESCAPES'
> 
> 1. make view printing sql_mode independent
>   1.1. Print always current escape character
>   1.2. Allow escape character usage independent of sql_mode
> 2. Add workaround for parsing "like 3 in (0,1) escape 3" problem
>  (alwaysuse parances in case of printing ESCAPE)
> 
> diff --git a/mysql-test/main/ctype_cp1250_ch.result 
> b/mysql-test/main/ctype_cp1250_ch.result
> index b0aa4cff382..88175ae5013 100644
> --- a/mysql-test/main/ctype_cp1250_ch.result
> +++ b/mysql-test/main/ctype_cp1250_ch.result
> @@ -129,7 +129,7 @@ EXPLAIN EXTENDED SELECT * FROM t1 WHERE CONCAT(c1)='a' 
> AND CONCAT(c1) LIKE 'a ';
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsfilteredExtra
>  1SIMPLE  t1  ALL NULLNULLNULLNULL2   100.00  
> Using where
>  Warnings:
> -Note 1003select `test`.`t1`.`c1` AS `c1` from `test`.`t1` where 
> concat(`test`.`t1`.`c1`) = 'a' and concat(`test`.`t1`.`c1`) like 'a '
> +Note 1003select `test`.`t1`.`c1` AS `c1` from `test`.`t1` where 
> concat(`test`.`t1`.`c1`) = 'a' and concat(`test`.`t1`.`c1`) like ('a ') 
> escape '\\'

I don't think there's a need to print the escape clause here, it only
makes the output less readable.

> diff --git a/mysql-test/main/default.result b/mysql-test/main/default.result
> index c9ac96367aa..5c72b894bf6 100644
> --- a/mysql-test/main/default.result
> +++ b/mysql-test/main/default.result
> @@ -1936,7 +1936,7 @@ CREATE OR REPLACE TABLE t1 ( col INT DEFAULT ( 1 LIKE ( 
> NOW() BETWEEN '2000-01-0
>  SHOW CREATE TABLE t1;
>  TableCreate Table
>  t1   CREATE TABLE `t1` (
> -  `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01' 
> and '2012-12-12'))
> +  `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01' 
> and '2012-12-12') escape '\\')

So, there're basically two approaches to a fix and we're inconsistently
using both for different items.

1. one can always print expressions in the environment-independent way
   - create longer less readable text
   - not all items support it, new syntax might be needed
 (cannot emulate NO_BACKSLASH_ESCAPES with the ESCAPE clause)

2. one can set the environment (sql_mode) to the same hard-coded value
   before parsing views and vcols
   - one cannot dump the view/table definition in one sql_mode and load
 in another
 + this isn't normally a problem, as mysqldump also uses a fixed
 sql_mode
   - some item properties (what are they?) might be only set at
 fix_fields, fixing sql_mode during parsing won't help there

Could we try to agree to one approach and use it consistently for all
items from now on?

> diff --git a/mysql-test/main/parser.result b/mysql-test/main/parser.result
> index 0bb4e82c8b8..79ec9f539eb 100644
> --- a/mysql-test/main/parser.result
> +++ b/mysql-test/main/parser.result
> @@ -1325,11 +1325,11 @@ select 1 between 2 in (3,4) and 5 AS `1 between (2 in 
> (3,4)) and 5`
>  create or replace view v1 as select 1 between (2 like 3) and 4;
>  Select view_definition from information_schema.views where 
> table_schema='test' and table_name='v1';
>  view_definition
> -select 1 between 2 like 3 and 4 AS `1 between (2 like 3) and 4`
> +select 1 between 2 like (3) escape '\\' and 4 AS `1 between (2 like 3) and 4`
>  create or replace view v1 as select 1 not between (2 like 3) and 4;
>  Select view_definition from information_schema.views where 
> table_schema='test' and table_name='v1';
>  view_definition
> -select 1 not between 2 like 3 and 4 AS `1 not between (2 like 3) and 4`
> +select 1 not between 2 like (3) escape '\\' and 4 AS `1 not between (2 like 
> 3) and 4`
>  drop view v1;
>  #
>  # Start of 10.2 tests
> diff --git a/mysql-test/main/precedence.result 
> b/mysql-test/main/precedence.result
> index fc6579651b4..ce26ca2acf2 100644
> --- a/mysql-test/main/precedence.result
> +++ b/mysql-test/main/precedence.result
> @@ -113,7 +113,7 @@ NOT 2 != 3NOT (2 != 3)(NOT 2) != 3
>  create or replace view v1 as select NOT 2 LIKE 3, NOT (2 LIKE 3), (NOT 2) 
> LIKE 3;
>  Select view_definition from information_schema.views where 
> table_schema='test' and table_name='v1';
>  view_definition
> -select 2  not like 3 AS `NOT 2 LIKE 3`,2  not like 3 AS `NOT (2 LIKE 3)`,!2 
> like 3 AS `(NOT 2) LIKE 3`
> +select 2  not like (3) escape '\\' AS `NOT 2 LIKE 3`,2  not like (3) escape 
> '\\' AS `NOT (2 LIKE 3)`,!2 like (3) escape '\\' AS `(NOT 2) LIKE 3`
>  select NOT 2 LIKE 3, NOT (2 LIKE 3), (NOT 2) LIKE 3 union select * from v1;
>  NOT 2 LIKE 3 NOT (2 LIKE 3)  (NOT 2) LIKE 3
>  11   0
> diff --git a/mysql-test/main/precedence_bugs.result 
> 

Re: [Maria-developers] 6ff49e48e4f: MDEV-26161: fix of view protocol

[Sergei Golubchik]

Hi, Oleksandr,

On Jan 06, Oleksandr Byelkin wrote:
> revision-id: 6ff49e48e4f (mariadb-10.3.37-54-g6ff49e48e4f)
> parent(s): 92ff948d021
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-12-06 10:22:06 +0100
> message:
> 
> MDEV-26161: fix of view protocol

could you please amend the commit comment to explain why
the statement produces a different error in a view protocol?

> diff --git a/mysql-test/main/gis.result b/mysql-test/main/gis.result
> index 358be520b06..37613bc71a0 100644
> --- a/mysql-test/main/gis.result
> +++ b/mysql-test/main/gis.result
> @@ -4980,11 +4980,11 @@ ERROR HY000: Illegal parameter data type geometry for 
> operation 'is_used_lock'
>  # MDEV-26161 crash in Gis_point::calculate_haversine
>  #
>  select st_distance_sphere(x'010300040004', 
> multipoint(point(124,204)), 10);
> -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY 
> field
> +Got one of the listed errors
>  select st_distance_sphere(x'010300040004', 
> multipoint(point(124,204)), 10);
> -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY 
> field
> +Got one of the listed errors
>  select st_distance_sphere(x'010300010004', 
> multipoint(point(124,204)), 10);
> -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY 
> field
> +Got one of the listed errors
>  #
>  # End of 10.3 tests
>  #
> diff --git a/mysql-test/main/gis.test b/mysql-test/main/gis.test
> index 716fab9bfeb..07b207dcdc9 100644
> --- a/mysql-test/main/gis.test
> +++ b/mysql-test/main/gis.test
> @@ -3093,11 +3093,11 @@ SELECT IS_USED_LOCK(POINT(1,1));
>  --echo #
>  --echo # MDEV-26161 crash in Gis_point::calculate_haversine
>  --echo #
> ---error ER_CANT_CREATE_GEOMETRY_OBJECT
> +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR
>  select st_distance_sphere(x'010300040004', 
> multipoint(point(124,204)), 10);
> ---error ER_CANT_CREATE_GEOMETRY_OBJECT
> +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR
>  select st_distance_sphere(x'010300040004', 
> multipoint(point(124,204)), 10);
> ---error ER_CANT_CREATE_GEOMETRY_OBJECT
> +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR
>  select st_distance_sphere(x'010300010004', 
> multipoint(point(124,204)), 10);
>  
>  --echo #
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 6a8268b7893: MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from mysql.proc. The table is probably corrupted"

[Sergei Golubchik]

Hi, Rucha,

On Dec 28, Rucha Deodhar wrote:
> revision-id: 6a8268b7893 (mariadb-10.6.11-62-g6a8268b7893)
> parent(s): 4ca5a0ec98d
> author: Rucha Deodhar
> committer: Rucha Deodhar
> timestamp: 2022-12-15 17:45:25 +0530
> message:
> 
> MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from
> mysql.proc. The table is probably corrupted"
> 
> Analysis: When mysql_upgrade runs statements for upgrade, characterset is
> converted to utf8mb4 because server starts with old_mode that interprets
> utf8 to utf8mb4, but mysql.proc table has "utf8mb3" as hardcoded, so
> it crashes with corrupted table.
> 
> Fix: Changed Table_check_intact::check() definition to allow both
> utf8mb3 and utf8mb4 by checking prefix and changing the upgrade scripts
> to explicitly use utf8mb3
> 
> diff --git a/mysql-test/main/mysql_upgrade-28915.opt 
> b/mysql-test/main/mysql_upgrade-28915.opt
> new file mode 100644
> index 000..0be5567a64d
> --- /dev/null
> +++ b/mysql-test/main/mysql_upgrade-28915.opt
> @@ -0,0 +1 @@
> +--old-mode=NO_DUP_KEY_WARNINGS_WITH_IGNORE

it's a bit confusing, try to use

--old-mode=

that is, empty old_mode, otherwise one starts thinking what this bug has
to do with NO_DUP_KEY_WARNINGS_WITH_IGNORE.

> diff --git a/mysql-test/main/mysql_upgrade-28915.test 
> b/mysql-test/main/mysql_upgrade-28915.test
> new file mode 100644
> index 000..61b1fc263cb
> --- /dev/null
> +++ b/mysql-test/main/mysql_upgrade-28915.test
> @@ -0,0 +1,78 @@
> +--source include/mysql_upgrade_preparation.inc
> +
> +let $MYSQLD_DATADIR= `select @@datadir`;
> +
> +--echo #
> +--echo # Beginning of 10.6 test
> +--echo #
> +--echo # MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot 
> load
> +--echo # from mysql.proc. The table is probably corrupted"
> +
> +--echo # Show that tables are created with utf8mb3 even without 
> UTF8_IS_UTF8MB3 (see the .opt file)

I don't think you've shown it. mtr does not recreate system tables for
every test. You need to force it to re-bootstrap with one of the options
that do it, for example, --innodb-page-size=16k (search mtr for
"rebootstrap" for details). But even that won't help, because it'll only
add "rebootstrap" options to the server's command line when
bootstrapping. May be a combination of .cnf and .opt can do what you
want, I didn't try.

> +
> +SHOW CREATE TABLE mysql.proc;
> +SHOW CREATE TABLE mysql.event;
> +
> +--exec $MYSQL_UPGRADE --force 2>&1
> +
> +--remove_file $MYSQLD_DATADIR/mysql_upgrade_info
> +
> +SHOW CREATE TABLE mysql.proc;
> +SHOW CREATE TABLE mysql.event;
> +
> +--echo # Emulate that tables were created with utf8mb4 by an older version
> +
> +ALTER TABLE mysql.proc MODIFY db CHAR(64) CHARACTER SET utf8mb4 COLLATE 
> utf8mb4_general_ci NOT NULL,
> +   MODIFY name CHAR(64) CHARACTER SET utf8mb4 COLLATE 
> utf8mb4_general_ci NOT NULL,
> +   MODIFY specific_name CHAR(64) CHARACTER SET utf8mb4 
> COLLATE utf8mb4_general_ci DEFAULT NULL,
> +   MODIFY definer VARCHAR(64) CHARACTER SET utf8mb4 
> COLLATE utf8mb4_general_ci DEFAULT NULL,
> +   MODIFY comment TEXT CHARACTER SET utf8mb4 COLLATE 
> utf8mb4_general_ci DEFAULT NULL,
> +   MODIFY character_set_client CHAR(32) CHARACTER SET 
> utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
> +   MODIFY collation_connection CHAR(32) CHARACTER SET 
> utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
> +   MODIFY db_collation CHAR(32) CHARACTER SET utf8mb4 
> COLLATE utf8mb4_general_ci DEFAULT NULL;
> +
> +ALTER TABLE mysql.event MODIFY db CHAR(64) CHARACTER SET utf8mb4 COLLATE 
> utf8mb4_bin NOT NULL DEFAULT '',
> +MODIFY name CHAR(64) CHARACTER SET utf8mb4 NOT NULL 
> DEFAULT '',
> +MODIFY definer VARCHAR(384) CHARACTER SET utf8mb4 
> COLLATE utf8mb4_bin NOT NULL DEFAULT '',
> +MODIFY comment CHAR(64) CHARACTER SET utf8mb4 
> COLLATE utf8mb4_bin NOT NULL DEFAULT '',
> +MODIFY character_set_client CHAR(32) CHARACTER SET 
> utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
> +MODIFY collation_connection CHAR(32) CHARACTER SET 
> utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
> +MODIFY db_collation CHAR(32) CHARACTER SET utf8mb4 
> COLLATE utf8mb4_bin DEFAULT NULL;
> +
> +--echo # mysql_upgrade changes columns from utf8mb4 to utf8mb3
> +
> +--exec $MYSQL_UPGRADE --force 2>&1
> +--remove_file $MYSQLD_DATADIR/mysql_upgrade_info
> +
> +--vertical_results
> +
> +SHOW CREATE TABLE mysql.proc;
> +SHOW CREATE TABLE mysql.event;
> +
> +CREATE TABLE t1 (id1 INT, val1 VARCHAR(5));
> +
> +DELIMITER |;
> +
> +CREATE PROCEDURE sp1 ()
> + BEGIN
> +  SELECT val1 FROM t1;
> + END|

You've made changes in sp.cc and event_db_repository.cc to ensure that
these tables are usable even with utf8mb4 charset.

But you don't test it if it works. You create a 

Re: [Maria-developers] 4ddf606debf: MDEV-30151 parse error 1=2 not between/in

[Sergei Golubchik]

Hi, Alexander,

I'm not sure I understand everything you did there, so questions below

On Dec 28, Alexander Barkov wrote:
> revision-id: 4ddf606debf (mariadb-10.3.37-59-g4ddf606debf)
> parent(s): 8f30973234d
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-12-13 10:35:16 +0400
> message:
> 
> MDEV-30151 parse error 1=2 not between/in
> 
> diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy
> index 1f6485dac6a..ca614ff63e9 100644
> --- a/sql/sql_yacc.yy
> +++ b/sql/sql_yacc.yy
> @@ -1687,7 +1687,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t 
> *yystacksize);
>  
>  %left   PREC_BELOW_NOT
>  
> -%nonassoc NOT_SYM
> +/* The precendence of boolean NOT is in fact here. See the comment below. */
> +
>  %left   '=' EQUAL_SYM GE '>' LE '<' NE
>  %nonassoc IS
>  %right BETWEEN_SYM
> @@ -1699,6 +1700,24 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t 
> *yystacksize);
>  %left   '*' '/' '%' DIV_SYM MOD_SYM
>  %left   '^'
>  %left   MYSQL_CONCAT_SYM
> +/*
> +  Boolean negation has a special branch in "expr" starting with NOT_SYM.
> +  The precedence of logical negation is determined by the grammar itself
> +  (without using Bison terminal symbol precedence) in this order
> +  - Boolean factor (i.e. logical AND)
> +  - Boolean NOT
> +  - Boolean test (such as '=', IS NULL, IS TRUE)
> +
> +  But we also need a precedence for NOT_SYM in other contexts,
> +  to shift (without reduce) in these cases:
> + predicate  NOT IN ...
> + predicate  NOT BETWEEN ...
> + predicate  NOT LIKE ...
> + predicate  NOT REGEXP ...
> +  If the precedence of NOT_SYM was low, it would reduce immediately
> +  after scanning "predicate" and then produce a syntax error on "NOT".
> +*/
> +%nonassoc NOT_SYM
>  %nonassoc NEG '~' NOT2_SYM BINARY
>  %nonassoc COLLATE_SYM
>  
> @@ -1938,6 +1957,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t 
> *yystacksize);
>  literal insert_ident order_ident temporal_literal
>  simple_ident expr sum_expr in_sum_expr
>  variable variable_aux
> +boolean_test boolean_factor
>  predicate bit_expr parenthesized_expr
>  table_wild simple_expr column_default_non_parenthesized_expr udf_expr
>  primary_expr string_factor_expr mysql_concatenation_expr
> @@ -9840,79 +9860,87 @@ expr:
>  MYSQL_YYABORT;
>  }
>}
> -| NOT_SYM expr %prec NOT_SYM
> +| boolean_factor

what does that achieve?

> +;
> +
> +boolean_factor:
> +  NOT_SYM boolean_factor
>{
>  $$= negate_expression(thd, $2);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS TRUE_SYM %prec IS
> +| boolean_test %prec PREC_BELOW_NOT
> +;
> +
> +boolean_test:
> +  boolean_test IS TRUE_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_istrue(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS not TRUE_SYM %prec IS
> +| boolean_test IS not TRUE_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_isnottrue(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS FALSE_SYM %prec IS
> +| boolean_test IS FALSE_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_isfalse(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS not FALSE_SYM %prec IS
> +| boolean_test IS not FALSE_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_isnotfalse(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS UNKNOWN_SYM %prec IS
> +| boolean_test IS UNKNOWN_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_isnull(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS not UNKNOWN_SYM %prec IS
> +| boolean_test IS not UNKNOWN_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_isnotnull(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS NULL_SYM %prec PREC_BELOW_NOT
> +| boolean_test IS NULL_SYM %prec PREC_BELOW_NOT

can you try %prec IS here?
I suspect that if this affects anything, it'll cause bugs

>{
>  $$= new (thd->mem_root) Item_func_isnull(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -| expr IS not NULL_SYM %prec IS
> +| boolean_test IS not NULL_SYM %prec IS
>{
>  $$= new (thd->mem_root) Item_func_isnotnull(thd, $1);
>  if (unlikely($$ == NULL))
>MYSQL_YYABORT;
>}
> -  

Re: [Maria-developers] 4d9f8a3c31e: MDEV-28669 addendum: additional tests and mtr changes

[Sergei Golubchik]

Hi, Julius,

On Dec 27, Julius Goryavsky wrote:

> commit 4d9f8a3c31e
> Author: Julius Goryavsky 
> Date:   Mon Dec 19 10:29:55 2022 +0100
> 
> diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl
> index 12afab4d28c..8df8198fc99 100755
> --- a/mysql-test/mysql-test-run.pl
> +++ b/mysql-test/mysql-test-run.pl
> @@ -313,7 +313,7 @@ my %mysqld_logs;
>  my $opt_debug_sync_timeout= 300; # Default timeout for WAIT_FOR actions.
>  my $warn_seconds = 60;
>  
> -my $rebootstrap_re= 
> '--innodb[-_](?:page[-_]size|checksum[-_]algorithm|undo[-_]tablespaces|log[-_]group[-_]home[-_]dir|data[-_]home[-_]dir)|data[-_]file[-_]path|force_rebootstrap';
> +my $rebootstrap_re= 
> '--(?:loose[-_])?(?:innodb[-_](?:page[-_]size|file(?:[-_]format|per[-_]table)|compression[-_](?:default|algorithm)|checksum(?:s|[-_]algorithm)|undo[-_](?:directory|tablespaces)|(?:data|log[-_]group)[-_]home[-_]dir|buffer[-_]pool[-_]filename|data[-_]file[-_]path|default[-_](?:encryption[-_]key[-_]id|page[-_]encryption)|sys[-_]|(?:index|table)[-_]stats)|aria[-_]log[-_](?:dir[-_]path|file[-_]size))|force_rebootstrap';

1. drop force_rebootstrap please
2. this code was already not very readable, but got fairly ridiculous
now. Let's change the approach. E.g. put them all in a hash, like

 my %rebootstrap_check = map { $_ => 1 } qw(
   innodb-page-size
   innodb-file-format
   ...
);

you'll need to normalize an option before looking it up in the hash

>  
>  sub testcase_timeout ($) { return $opt_testcase_timeout * 60; }
>  sub check_timeout ($) { return testcase_timeout($_[0]); }
> @@ -2762,7 +2762,29 @@ sub mysql_server_start($) {
>  return;
>}
>  
> -  my $datadir= $mysqld->value('datadir');
> +  my $extra_opts= get_extra_opts($mysqld, $tinfo);
> +
> +  # The test options can contain the --datadir parameter, but
> +  # the bootstrap function can only read datadir location from
> +  # a .cnf file. So we need to parse the options to get the
> +  # actual location of the data directory, considering both
> +  # the options and the .cnf file, and then store the resulting
> +  # value in a "$mysqld" hash - to use later, when we need to
> +  # know the actual location of the data directory:
> +  my $datadir="";
> +  foreach my $opt ( @$extra_opts )
> +  {
> +if ($opt =~ /^--datadir=/) {
> +  $datadir = substr($opt, 10);
> +  last;

strictly speaking, `last` is incorrect, there can be more than
one --datadir

> +}
> +  }
> +  # If datadir is not in the options, then read it from .cnf:
> +  if (! $datadir) {
> +$datadir = $mysqld->value('datadir');
> +  }
> +  $mysqld->{'datadir'} = $datadir;
> +
>if (not $opt_start_dirty)
>{
>  
> @@ -2785,17 +2807,59 @@ sub mysql_server_start($) {
>  }
>}
>  
> +  # Run -master.sh
> +  if ($mysqld->option('#!run-master-sh') and
> +  defined $tinfo->{master_sh} and
> +  run_system('/bin/sh ' . $tinfo->{master_sh}) )
> +  {
> +$tinfo->{'comment'}= "Failed to execute '$tinfo->{master_sh}'";
> +return 1;
> +  }
> +
> +  # Run -slave.sh
> +  if ($mysqld->option('#!run-slave-sh') and
> +  defined $tinfo->{slave_sh} and
> +  run_system('/bin/sh ' . $tinfo->{slave_sh}))
> +  {
> +$tinfo->{'comment'}= "Failed to execute '$tinfo->{slave_sh}'";
> +return 1;
> +  }
> +
>my $mysqld_basedir= $mysqld->value('basedir');
> -  my $extra_opts= get_extra_opts($mysqld, $tinfo);
>  
>if ( $basedir eq $mysqld_basedir )
>{
>  if (! $opt_start_dirty)  # If dirty, keep possibly grown system db
>  {
> +  # Find the name of the current section in the configuration
> +  # file and its suffix:
> +  my $section = $mysqld->{name};
> +  my $server_name;
> +  my $suffix = "";
> +  if (index($section, '.') != -1) {
> +($server_name, $suffix) = $section =~ 
> /^\s*([^\s.]+)\s*\.\s*([^\s]+)/;
> +  }
> +  else {
> +$server_name = $section =~ /^\s*([^\s]+)/;
> +  }

Replace all 11 lines above with:

 my ($suffix) = ($mysqld->{name} =~ /\.(.*)/);

># Some InnoDB options are incompatible with the default bootstrap.
># If they are used, re-bootstrap
>my @rebootstrap_opts;
>@rebootstrap_opts = grep {/$rebootstrap_re/o} @$extra_opts if 
> $extra_opts;
> +  # Let's store the additional bootstrap options in
> +  # the environment variable - they may be used later
> +  # in the mtr tests - for re-bootstrap or run the
> +  # recovery, etc:
> +  my $extra_text = "--datadir=$datadir";

that's rather random. Why do you add only this one option, that doesn't
make sense. Better handle any additional requirements in the test.

> +  if (@rebootstrap_opts) {

why do you need an if() ?

> +$extra_text .= ' '.join(' ', @rebootstrap_opts);
> +  }
> +  if ($suffix) {
> +$ENV{'MTR_BOOTSTRAP_OPTS_'.$suffix} = $extra_text;
> +  }
> +  else {
> +$ENV{'MTR_BOOTSTRAP_OPTS'} = $extra_text;

Is that even possible?

> +  }
>  

Re: [Maria-developers] f984f49fba7: MDEV-30203 - deb fix piuparts

[Sergei Golubchik]

Hi, Daniel,

On Dec 20, Daniel Black wrote:
> updated as bb-11.0-danielblack-MDEV-30203-pkgtest-symlink-packages on top
> of bb-11.0-serg branch

please, rebase again (your three commits adjusting tests to 11.0 will
disappear). Also, see below:

> > > diff --git a/cmake/symlinks.cmake b/cmake/symlinks.cmake
> > > index 3f3b4e4a9b5..a4eb6c2b42c 100644
> > > --- a/cmake/symlinks.cmake
> > > +++ b/cmake/symlinks.cmake
> > > @@ -12,7 +12,6 @@ endmacro()
> > >  REGISTER_SYMLINK("mariadb" "mysql")
> > >  REGISTER_SYMLINK("mariadb-access" "mysqlaccess")
> > >  REGISTER_SYMLINK("mariadb-admin" "mysqladmin")
> > > -REGISTER_SYMLINK("mariadb-backup" "mariabackup")
> >
> > better not. there're scripts that might be using it.
> > This is why I suggested `strncmp(my_progname, "mariadb", 7)`
> > and not `strncmp(my_progname, "maria", 5)` in your PR
> 
> ok. Moved to direct link creation as REGISTGER_SYMLINK is used to
> determine compat links.

Why not to use compat links? I mean, you create compat links for the
test component and manually list it for

+SET(CPACK_COMPONENT_TESTSYMLINKS_GROUP "test")

so it doesn't create MariaDB-test-compat. This would work for backup
too, wouldn't it?

> sure, lets try - pushed
> bb-11.0-danielblack-MDEV-30203-pkgtest-symlink-packages
> 
> https://salsa.debian.org/grooverdan/mariadb-server/-/pipelines/472980

apparently salsa was unhappy with it. "missing breaks".
does it mean we have to keep that?

> > > --- a/debian/mariadb-backup.install
> > > +++ b/debian/mariadb-backup.install
> > > @@ -1,6 +1,4 @@
> > > -usr/bin/mariabackup
> >
> > same here. needs mariadb-backup-compat
> 
> for the time being I've kept mariadb-backup as the only package (no
> mariadb-backup-compat) and it still has the exe symlink
> 
> > >  usr/bin/mariadb-backup
> > >  usr/bin/mbstream
> > > -usr/share/man/man1/mariabackup.1
> 
> but this man link is removed.

why did you remove this man symlink? I thought Debian is pretty strict
on having manpages for everything

> > you forgot to move symlinks to man pages
> 
> opps. Late push coming.

please, squash it with the main commit, when rebasing

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] f984f49fba7: MDEV-30203 - deb fix piuparts

[Sergei Golubchik]

Hi, Daniel,

This is a review of `git diff 12786f0e779f 5f76981eb649`, that is, for
commits:

5f76981eb64 rpm: packages that dont match component name
25eaa0aebfa MDEV-30205: [Cmtr: suite look in mariadb-test path
0981201f027 mtr - wsrep use mariadb executables
eba0b35035a mtr: maradb names for *embedded and mariadb-client-test
974ca88f0fe deb: fix autotest - exclude for MDEV-30205
b19d75408bb Deb: Breaks/Replaces/COnflicts
c60f0341950 MDEV-30203 deb breaks/replaces on compat packages
f984f49fba7 MDEV-30203 - deb fix piuparts
310d025ff0d MDEV-30205: fix deb autopkgtest
89f6cf47ee2 rpm: fix packaging of compat/non-compat scripts links
494ed76d15a deb: mysqlhotcopy to mariadb-client-compat
d03075f3a1f deb: server-core mysql* -> server-compat
9453812193d wsrep_sst_mariabackup to use mariadb-backup (and message 
accordingly)
a4f843a7a5a Fix uncollected symlink
eb9a469bf9a Add package recommended dependencies
c135be62274 Remove duplicate entry
84c7cd29c61 Fix Debian packaging for mariabackup
5dbc7ae8bb4 MDEV-30203: compat packages - RPM  remove mariabackup link
eb4fa5d750f MDEV-30203: debian - make compat packages
d4be0b9be73 MDEV-30203: Create mysql symlink packages (RPM) - scripts
441879fd493 MDEV-30203: Create mysql symlink packages (RPM)

On Dec 19, Daniel Black wrote:

> diff --git a/cmake/cpack_rpm.cmake b/cmake/cpack_rpm.cmake
> index 339363b2169..7b48fd64f4c 100644
> --- a/cmake/cpack_rpm.cmake
> +++ b/cmake/cpack_rpm.cmake
> @@ -85,6 +85,35 @@ SET(CPACK_RPM_server_PACKAGE_DESCRIPTION 
> "${CPACK_RPM_PACKAGE_DESCRIPTION}")
>  SET(CPACK_RPM_test_PACKAGE_SUMMARY "MariaDB database regression test suite")
>  SET(CPACK_RPM_test_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}")
>  
> +# "set/append array" - append a set of strings, separated by a space
> +MACRO(SETA var)
> +  FOREACH(v ${ARGN})
> +SET(${var} "${${var}} ${v}")
> +  ENDFOREACH()
> +ENDMACRO(SETA)
> +
> +FOREACH(SYM_COMPONENT Server Client Test)
> +  SET(SYM ${SYM_COMPONENT}_symlinks)
> +  string(TOLOWER ${SYM} SYM)

* why all cmake commands are uppercase, but `string` is lowercase?
* SET is redundant, you can do

  STRING(TOLOWER ${SYM_COMPONENT}_symlinks SYM)

* may be, better to use ${SYM_COMPONENT}-symlinks ?
  Like in MariaDB-server-symlinks ?

> +  SET(SYMCOMP ${SYM_COMPONENT}Symlinks)
> +  string(TOUPPER ${SYMCOMP} SYMCOMP_UPPER)
> +  SET(CPACK_COMPONENT_${SYMCOMP_UPPER}_GROUP "${SYM}")
> +  SET(CPACK_COMPONENTS_ALL "${CPACK_COMPONENTS_ALL}" "${SYMCOMP}")
> +  SET(CPACK_RPM_${SYM}_PACKAGE_SUMMARY "MySQL compatible symlinks for 
> MariaDB database ${SYM_COMPONENT} binaries/scripts")
> +  SET(CPACK_RPM_${SYM}_PACKAGE_DESCRIPTION 
> "${CPACK_RPM_PACKAGE_DESCRIPTION}")
> +  SET(CPACK_RPM_${SYM}_PACKAGE_ARCHITECTURE "noarch")
> +  SETA(CPACK_RPM_${SYM_COMPONENT}_PACKAGE_RECOMMENDS "MariaDB-${SYM}")

You forgot to say that symlink package DEPENDS on the non-symlink package

> +ENDFOREACH()
> +
> +# TODO change to 11.0 on rebase
> +SETA(CPACK_RPM_client_symlinks_PACKAGE_CONFLICTS
> +  "MariaDB-client < 10.11.2"
> +  "MariaDB-server < 10.11.2")
> +SETA(CPACK_RPM_server_symlinks_PACKAGE_CONFLICTS
> +  "MariaDB-server < 10.11.2")
> +SETA(CPACK_RPM_test_symlinks_PACKAGE_CONFLICTS
> +  "MariaDB-test < 10.11.2")
> +
>  # libmariadb3
>  SET(CPACK_RPM_shared_PACKAGE_SUMMARY "LGPL MariaDB database client library")
>  SET(CPACK_RPM_shared_PACKAGE_DESCRIPTION "This is LGPL MariaDB client 
> library that can be used to connect to MySQL
> diff --git a/cmake/symlinks.cmake b/cmake/symlinks.cmake
> index 3f3b4e4a9b5..a4eb6c2b42c 100644
> --- a/cmake/symlinks.cmake
> +++ b/cmake/symlinks.cmake
> @@ -12,7 +12,6 @@ endmacro()
>  REGISTER_SYMLINK("mariadb" "mysql")
>  REGISTER_SYMLINK("mariadb-access" "mysqlaccess")
>  REGISTER_SYMLINK("mariadb-admin" "mysqladmin")
> -REGISTER_SYMLINK("mariadb-backup" "mariabackup")

better not. there're scripts that might be using it.
This is why I suggested `strncmp(my_progname, "mariadb", 7)`
and not `strncmp(my_progname, "maria", 5)` in your PR

>  REGISTER_SYMLINK("mariadb-binlog" "mysqlbinlog")
>  REGISTER_SYMLINK("mariadb-check" "mysqlcheck")
>  REGISTER_SYMLINK("mariadb-client-test-embedded" "mysql_client_test_embedded")
> diff --git a/debian/control b/debian/control
> index 81418c13656..6b225ca4b3d 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -548,7 +548,8 @@ Provides: default-mysql-client,
>virtual-mysql-client
>  Recommends: libdbd-mariadb-perl | libdbd-mysql-perl,
>  libdbi-perl,
> -libterm-readkey-perl
> +libterm-readkey-perl,
> +mariadb-client-compat

Please, no. deb and rpm packages are different enough already.
don't use different suffixes for them.
Choose either -symlink or -compat (or something else) and use it
consistently both in deb and in rpm.

>  Description: MariaDB database client binaries
>   MariaDB is a fast, stable and true multi-user, multi-threaded SQL database
>   server. SQL (Structured Query Language) is the most 

Re: [Maria-developers] MDEV-14443 test cases only

[Sergei Golubchik]

Hi, Vicențiu,

This is a review of _test cases only_.
Mainly, to discuss the behavior.

Random thoughts here and comments inline in the diff.

There are two models for DENY that I can think of: "a hole in grants"
and "a mask".

The difference lies in where DENY is applied. In the "a hole in grants"
model one has a certain set of grants (may be with "holes"), a role has
another set of grants, and when one does SET ROLE, one gets a union of
those grants. Like, one has SELECT on db1.* and DENY to db1.t1, and
some role has SELECT on db1.t1 - by setting this role a user will have
full SELECT on db1.*

While in the "mask" model, denies are separate entities, when setting a
role a union of denies is subtracted from the union of grants. In the
example above a user would still be denied access to db1.t1.

I personally see the "hole" model as more logical and easier to
understand. It also allows to "patch" holes with roles, what the "mask"
model cannot do.

The "mask" model needs getting used to, the fact that after SET ROLE one
can end up having less access than before. It's not clear who can DENY
what and to whom. And DENY is very difficult to work around, esp. as one
can DENY IGNORE DENIES.

But it allows to DENY something to PUBLIC, and the "hole" model cannot
really do that. And I think it'd be a popular use case.

Ideally, I'd prefer to have some kind of a compromise model that's as
simple to understand as a "hole", but allows to deny to public.
Unfortunately, I don't see how to do that.

You've implemented the "mask" behavior. Okay. It solves an important use
case.

More comments below:

> diff --git a/mysql-test/suite/deny/columns.result 
> b/mysql-test/suite/deny/columns.result
> --- /dev/null
> +++ b/mysql-test/suite/deny/columns.result
> @@ -0,0 +1,126 @@
> +create user foo;
> +create database deny_db;
> +create table deny_db.t1 (a int, b int, secret int);
> +create table deny_db.t2 (a2 int, b2 int, secret2 int);
> +insert into deny_db.t2 values (100, 200, 300);
> +grant all on *.* to foo;
> +revoke ignore denies on *.* from foo;
> +grant all on deny_db.* to foo;
> +grant all on deny_db.t1 to foo;
> +grant all on deny_db.t2 to foo;
> +grant select (a, b, secret) on deny_db.t1 to foo;
> +grant insert (a, b, secret) on deny_db.t1 to foo;
> +grant update (a, b, secret) on deny_db.t1 to foo;
> +grant references (a, b, secret) on deny_db.t1 to foo;
> +grant select (a2, b2, secret2) on deny_db.t2 to foo;
> +grant insert (a2, b2, secret2) on deny_db.t2 to foo;
> +grant update (a2, b2, secret2) on deny_db.t2 to foo;
> +grant references (a2, b2, secret2) on deny_db.t2 to foo;

this is quite confusing, tons of redundant grants on different levels,
it makes it very unclear of what causes what.

> +#
> +# Test select and insert ... select commands.
> +#
> +deny select (secret) on deny_db.t1 to foo;
> +connect  con1,localhost,foo,,;
> +select * from deny_db.t1;
> +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 
> 'secret' in table 't1'

case in point. Normally, this should've been

  SELECT command denied to user 'foo'@'localhost' for table 't1'

like in

   create database db;
   create table db.t1 (a int, b int, secret int);
   insert db.t1 values (1,2,100);
   create user foo@localhost;
   grant select (a,b) on db.t1 to foo@localhost;
   connect foo,localhost,foo;
   select a,b from db.t1;
   --error ER_TABLEACCESS_DENIED_ERROR
   select * from db.t1;
   --error ER_COLUMNACCESS_DENIED_ERROR
   select a,b,secret from db.t1;

that is `select *` is ER_TABLEACCESS_DENIED_ERROR, and if I mention
the column by name, then it's ER_COLUMNACCESS_DENIED_ERROR

But you have so many redundant overlapping grants, that I really
cannot see what error is correct in your test here.

> +select secret from deny_db.t1;
> +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 
> 'secret' in table 't1'
> +#
> +# These are different code paths. Check both.
> +#
> +insert into deny_db.t1 (a, b, secret) values (1, 2, 3);
> +insert into deny_db.t1 values (1, 2, 3);
> +#
> +# Check INSERT SELECT as we need to account for 2 different kinds of
> +# privileges.
> +#
> +insert into deny_db.t1 select * from deny_db.t1;
> +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 
> 'secret' in table 't1'
> +insert into deny_db.t1 select * from deny_db.t2;
> +show full columns from deny_db.t1;
> +FieldTypeCollation   NullKey Default Extra   
> Privileges  Comment
> +aint(11) NULLYES NULL
> select,insert,update,references 
> +bint(11) NULLYES NULL
> select,insert,update,references 
> +secret   int(11) NULLYES NULL
> insert,update,references

do you have a test where a denied column isn't shown by SHOW COLUMNS?
that is, where you don't have other privileges on it?

> +show full columns from deny_db.t2;
> +FieldTypeCollation   NullKey 

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys

[Sergei Golubchik]

Hi, Nikita,

On Nov 01, Nikita Malyavin wrote:
> On Mon, 31 Oct 2022 at 21:22, Sergei Golubchik  wrote:
> 
> > > I hope RPL_TABLE_LIST can be preserved between the events. Is it
> > > so?
> >
> > Yes, it appears to be created in Table_map_log_event and removed on
> > commit. That is, it survives multiple row events.
> >
> > Still, there are only two possibilities:
> >
> > * the table has a PK - then row format doesn't matter, just use a PK
> > * the table has no PK - then row format doesn't matter, row pre-image
> >   will be always FULL
> 
> Sooo we can always just use the first key we have, assuming PK always
> comes first..

Not quite. If there is no PK, we should use the key with the highest
cardinality.

> > So, you're right that in the replication case it's also enough to
> > calculate the best key once, store it in RPL_TABLE_LIST, and not
> > recalculate it per event.
> >
> > But even then it doesn't need a complex cache, but just one
> >
> >   uint key_to_use;
> >
> > in RPL_TABLE_LIST.
> 
> Agree, one key is enough to store.
> 
> I also wanted to ask:
> -- What if ROW_FORMAT was changed in between of a transaction?

Doesn't matter, see above. If the table had PK, then simpy use PK, even
if ROW_FORMAT changes. If the table had no PK, then binlog will always
have the full row, no matter how you'll be changing ROW_FORMAT.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys

[Sergei Golubchik]

Hi, Nikita,

On Oct 31, Nikita Malyavin wrote:
> On Mon, 31 Oct 2022 at 14:08, Sergei Golubchik  wrote:
> > On Oct 31, Nikita Malyavin wrote:
> > >
> > > > Also, I've looked at your latest branch. What were you
> > > > optimizing with the commit 3afa3288221 (the one with
> > > > usable_keys_data)?
> > > >
> > > > It's complex and I don't quite see what's the purpose of it. It
> > > > looks like all you need to do is to decide on the best index to
> > > > use, once, save it somewhere, e.g. in RPL_TABLE_LIST, and that's
> > > > it.
> > >
> > > > This can be done, for example, in copy_data_between_tables().
> > > >
> > > For ONLINE ALTER TABLE yes, but what about usual replication? I'd
> > > prefer one generic algorithm for everything.
> >
> > For replication it already works.
> 
> It didn't work for cases with extra columns, when master a primary
> key, or some other key was amended with an extra column, for example.
> Or when the set of keys has been changed some other way.

Yes, replication didn't support those cases. It didn't look like users
care, though.

> > The key is found once, before a batch of rows. It cannot be done
> > less often than that, because the next row event can be for a
> > different table and have a different row format.
>
> I hope RPL_TABLE_LIST can be preserved between the events. Is it so?

Yes, it appears to be created in Table_map_log_event and removed on
commit. That is, it survives multiple row events.

Still, there are only two possibilities:

* the table has a PK - then row format doesn't matter, just use a PK
* the table has no PK - then row format doesn't matter, row pre-image
  will be always FULL

So, you're right that in the replication case it's also enough to
calculate the best key once, store it in RPL_TABLE_LIST, and not
recalculate it per event.

But even then it doesn't need a complex cache, but just one

  uint key_to_use;

in RPL_TABLE_LIST.

You only need some complex logic if the table has PK on the master, row
image is not FULL, and you cannot use this PK on the slave. This use
case wasn't supported before, it's not needed for online alter, so I
suggest we won't try to do it together with already big and complex
online alter task.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys

[Sergei Golubchik]

Hi, Nikita,

On Oct 31, Nikita Malyavin wrote:
> 
> > Also, I've looked at your latest branch. What were you optimizing
> > with the commit 3afa3288221 (the one with usable_keys_data)?
> >
> > It's complex and I don't quite see what's the purpose of it. It
> > looks like all you need to do is to decide on the best index to use,
> > once, save it somewhere, e.g. in RPL_TABLE_LIST, and that's it.
> 
> > This can be done, for example, in copy_data_between_tables().
> >
> For ONLINE ALTER TABLE yes, but what about usual replication? I'd
> prefer one generic algorithm for everything.

For replication it already works. The key is found once, before a batch
of rows. It cannot be done less often than that, because the next row
event can be for a different table and have a different row format.

For online alter all events are for the same table and all events have
full rows. There is no need to recalculate the key per event.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys

[Sergei Golubchik]

Hi, Nikita,

On Oct 18, Nikita Malyavin wrote:
> On Tue, 18 Oct 2022 at 01:01, Sergei Golubchik  wrote:
> 
> > > > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> > > > > index 422d496d5c3..25705d13641 100644
> > > > > --- a/sql/log_event_server.cc
> > > > > +++ b/sql/log_event_server.cc
> > > > > @@ -6062,7 +6065,7 @@ int 
> > > > > Rows_log_event::do_apply_event(rpl_group_info *rgi)
> > > > >  */
> > > > >  sql_mode_t saved_sql_mode= thd->variables.sql_mode;
> > > > >  if (!is_auto_inc_in_extra_columns())
> > > > > -  thd->variables.sql_mode= (rpl_data.is_online_alter() ? 
> > > > > saved_sql_mode :0)
> > > > > +  thd->variables.sql_mode= (m_online_alter ? saved_sql_mode :0)
> > > > > | MODE_NO_AUTO_VALUE_ON_ZERO;
> > > >
> > No, I didn't mean to ask "why MODE_NO_AUTO_VALUE_ON_ZERO", I meant to
> > ask why not
> >
> >   thd->variables.sql_mode= saved_sql_mode | MODE_NO_AUTO_VALUE_ON_ZERO;
> >
> > that is why not "to use saved_sql_mode also in normal replication"
> >
> Oh, sorry for misinterpretation. I have read what I wanted to read:)
> 
> Meanwhile I had a time to think. Imo it should be so in replication. It
> is a background process, right? So we have no control over sql_mode in
> the replica nodes (if I understand the design correctly).
> 
> So we should make it as permissive as possible. I have no opinion on
> NO_ZERO_DATE and other "View" (in terms of Model-View-Controller) modes,
> like EMPTY_STRING_IS_NULL, if they don't affect storage, they are
> unimportant. For the rest, we should allow as much execution, and as
> much conversions as possible, which is, as I recall, is achieved by
> zeroing the mode, and  MODE_NO_AUTO_VALUE_ON_ZERO is an exception that
> was found once.

Yes, agree. You're right.

> > > > eh?
> > > > 1. why it's *in the loop* ?
> > > > 2. why not simply
> > > >
> > > >   || f == f->table->next_number_field
> > > >
> > > >or, to be extra safe
> > > >
> > > >   || f->table->field[f->field_index] == f->table->next_number_field
> > > >
> > > Thanks!
> > > Turns out there's no need in extra safety, it just works.
> >
> > by "extra safety" I meant the case when
> >
> >   f != f->table->field[f->field_index]
> >
> > (here f= key->key_part[p].field). This is the case for varchar/blob
> > prefix keys. But as they cannot be auto-increment anyway, the direct
> > comparison should work, I suppose.
> 
> Hmm, thanks for noticing, I really didn't consider this blobs case.

still, you didn't use `f == f->table->next_number_field`
but we concluded it's safe, didn't we?

===

Also, I've looked at your latest branch. What were you optimizing with
the commit 3afa3288221 (the one with usable_keys_data)?

It's complex and I don't quite see what's the purpose of it.
It looks like all you need to do is to decide on the best index to
use, once, save it somewhere, e.g. in RPL_TABLE_LIST, and that's it.
This can be done, for example, in copy_data_between_tables().

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b2800c06b22: MDEV-28545 MyISAM reorganize partition corrupt older table format

[Sergei Golubchik]

Hi, Alexander,

Ok to push.
One minor comment below:

On Oct 25, Alexander Barkov wrote:
> revision-id: b2800c06b22 (mariadb-10.4.26-45-gb2800c06b22)
> parent(s): 34163154077
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-10-25 11:53:39 +0400
> message:
> 
> MDEV-28545 MyISAM reorganize partition corrupt older table format
> diff --git a/sql/sql_table.cc b/sql/sql_table.cc
> index 6aae927800e..a621825c395 100644
> --- a/sql/sql_table.cc
> +++ b/sql/sql_table.cc
> @@ -10068,6 +10068,18 @@ do_continue:;
>  
>set_table_default_charset(thd, create_info, alter_ctx.db);
>  
> +
> +#ifdef WITH_PARTITION_STORAGE_ENGINE
> +  if (!fast_alter_partition)
> +#endif

To avoid ifdef's you can use 

  if (IF_PARTITIONING(!fast_alter_partition, 1))

> +  {
> +/*
> +  We cannot alter partitions and change column data types at the same
> +  time.
> +*/
> +Create_field::upgrade_data_types(alter_info->create_list);
> +  }
> +
>if (create_info->check_fields(thd, alter_info,
>  table_list->table_name, table_list->db) ||
>create_info->fix_period_fields(thd, alter_info))
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 152295d1cb1: rpl: extra DEFAULTs should not be UPDATEd

[Sergei Golubchik]

Hi, Nikita,

As we've discussed on slack, this is wrong. The (serialized) behavior
should be as if all DML happened before ALTER.

Because if some DML would've happened after ALTER, it would be able to
see the new table definition and refer to new columns.

On Oct 24, Nikita Malyavin wrote:
> revision-id: 152295d1cb1 (mariadb-10.6.1-525-g152295d1cb1)
> parent(s): 90274a7428e
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2022-10-14 23:17:59 +0300
> message:
> 
> rpl: extra DEFAULTs should not be UPDATEd
> 
> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> index 25705d13641..a72005d70ab 100644
> --- a/sql/log_event_server.cc
> +++ b/sql/log_event_server.cc
> @@ -8832,11 +8832,17 @@ Update_rows_log_event::do_exec_row(rpl_group_info 
> *rgi)
>message= thd->wsrep_info;
>  #endif /* WSREP_PROC_INFO */
>  
> -  /* this also updates m_curr_row_end */
>thd_proc_info(thd, message);
> -  if (unlikely((error= unpack_current_row(rgi, _cols_ai
> +  Field **default_field= m_table->default_field;
> +  m_table->default_field= NULL;
> +  /* this also updates m_curr_row_end */
> +  error= unpack_current_row(rgi, _cols_ai);
> +  m_table->default_field= default_field;
> +
> +  if (unlikely(error))
>  goto err;
>  

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 39e20ca7e28: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations

[Sergei Golubchik]

Hi, Oleksandr,

On Oct 21, Oleksandr Byelkin wrote:
> > > > >
> > > > >if (share->view_def->parse((uchar *) >tabledef_version, 
> > > > > NULL,
> > > > >   view_timestamp_parameters, 1,
> > > > >   _parser_dummy_hook))
> > > > > +  {
> > > > > +// safety if the definition file is brocken
> > > > > +share->tabledef_version.length= 0;
> 
> if we got an error as I understood it will nead to an error of opening
> the table, so the query will  fail.
> 
> The other thread may have memory but it is true for any EOM situation
> and there are a lot of places where we allocate memory, so I do not
> see how this will differ.
> 
> The same with changing frm file with a working server, I do not think
> that our server will survive if something out of the server will
> change frms.

May be not, but it doesn't mean we need to introduce intentionally more 
places that can cause a crash. What will happen if you would not set

  share->tabledef_version.length= 0

there?

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 99dcf061c8c: MDEV-29481 mariadb-upgrade prints confusing statement

[Sergei Golubchik]

Hi, Alexander,

ok to push!

On Oct 21, Alexander Barkov wrote:
> revision-id: 99dcf061c8c (mariadb-10.4.26-55-g99dcf061c8c)
> parent(s): 1be451ca797
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-10-21 14:31:02 +0400
> message:
> 
> MDEV-29481 mariadb-upgrade prints confusing statement
> 
> This is a new version of the patch instead of the reverted:
> 
>   MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade
> 
> Ignore the difference in key packing flags HA_BINARY_PACK_KEY and HA_PACK_KEY
> during ALTER to allow ALGORITHM=INSTANT and ALGORITHM=NOCOPY in more cases.
> 
> If for some reasons (e.g. due to a bug fix such as MDEV-20704) these
> cumulative (over all segments) flags in KEY::flags are different for
> the old and new table inside compare_keys_but_name(), the difference
> in HA_BINARY_PACK_KEY and HA_PACK_KEY in KEY::flags is not really important:
> 
> MyISAM and Aria can handle such cases well: per-segment flags are stored in
> MYI and MAI files anyway and they are read during ha_myisam::open()
> ha_maria::open() time. So indexes get opened with correct per-segment
> flags that were calculated during the table CREATE time, no matter
> what the old (CREATE time) and new (ALTER TIME) per-index compression
> flags are, and no matter if they are equal or not.
> 
> All other engine ignore key compression flags, so this change
> is safe for other engines as well.
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 39e20ca7e28: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations

[Sergei Golubchik]

Hi, Oleksandr,

On Oct 19, Oleksandr Byelkin wrote:
> On Wed, Oct 19, 2022 at 5:09 PM Sergei Golubchik  wrote:
> > On Oct 19, Oleksandr Byelkin wrote:
> > >
> > >if (share->view_def->parse((uchar *) >tabledef_version, NULL,
> > >   view_timestamp_parameters, 1,
> > >   _parser_dummy_hook))
> > > +  {
> > > +// safety if the definition file is brocken
> > > +share->tabledef_version.length= 0;
> >
> > when can view_def->parse() fail? _Only_ if the definition is invalid?
> > Can it succeed for one TABLE and fail for another, both of the same
> > TABLE_SHARE?
> 
> Also if there is no memory.
> 
> If for the other table there will be more memory or file has changed
> it can succeed.

My point is that if it is possible that two different threads would get
different results from share->view_def->parse(), then you cannot update
the shared share->tabledef_version.length without a lock. And you cannot
read it without a lock.

So if it is possible that two different threads would get different
results from share->view_def->parse(), then it's best not to update
share->tabledef_version.length here and only do it in open_table_def().
 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 39e20ca7e28: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations

[Sergei Golubchik]

Hi, Oleksandr,

Looks good.
Just one question below:

On Oct 19, Oleksandr Byelkin wrote:
> revision-id: 39e20ca7e28 (mariadb-10.3.36-107-g39e20ca7e28)
> parent(s): d6707ab11f6
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-10-18 16:38:09 +0200
> message:
> 
> MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent 
> view operations
> 
> Read the version of the view share when we read definition to prevent
> simultaniouse access to a view table SHARE (and so its MEM_ROOT)
> from different threads.
> 
> diff --git a/sql/sql_view.cc b/sql/sql_view.cc
> index 68a8410f559..1e4a740f871 100644
> --- a/sql/sql_view.cc
> +++ b/sql/sql_view.cc
> @@ -1175,19 +1175,24 @@ static int mysql_register_view(THD *thd, TABLE_LIST 
> *view,
>  bool mariadb_view_version_get(TABLE_SHARE *share)
>  {
>DBUG_ASSERT(share->is_view);
> +  DBUG_ASSERT(share->tabledef_version.length == 0);
>  
>if (!(share->tabledef_version.str=
>  (uchar*) alloc_root(>mem_root,
>  MICROSECOND_TIMESTAMP_BUFFER_SIZE)))
>  return TRUE;
> -  share->tabledef_version.length= 0; // safety if the drfinition file is 
> brocken
>  
>DBUG_ASSERT(share->view_def != NULL);
>if (share->view_def->parse((uchar *) >tabledef_version, NULL,
>   view_timestamp_parameters, 1,
>   _parser_dummy_hook))
> +  {
> +// safety if the definition file is brocken
> +share->tabledef_version.length= 0;

when can view_def->parse() fail? _Only_ if the definition is invalid?
Can it succeed for one TABLE and fail for another, both of the same
TABLE_SHARE?

>  return TRUE;
> +  }
>DBUG_ASSERT(share->tabledef_version.length == 
> MICROSECOND_TIMESTAMP_BUFFER_SIZE-1);
> +
>return FALSE;
>  }

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys

[Sergei Golubchik]

Hi, Nikita,

On Oct 17, Nikita Malyavin wrote:
> Hello Sergei!
> 
> On Tue, 4 Oct 2022 at 21:02, Sergei Golubchik  wrote:
> 
> > Hi, Nikita,
> >
> > This review applies to the combined diff e2f8dff^..52f489e
> >
> > On Oct 04, Nikita Malyavin wrote:
> >
> > > diff --git a/sql/log_event.h b/sql/log_event.h
> > > index 91a910f..8452e665c98 100644
> > > --- a/sql/log_event.h
> > > +++ b/sql/log_event.h
> > > @@ -5275,8 +5278,11 @@ class Rows_log_event : public Log_event
> > >uchar*m_key;  /* Buffer to keep key value during searches */
> > >KEY  *m_key_info; /* Pointer to KEY info for m_key_nr */
> > >uint  m_key_nr;   /* Key number */
> > > +  uint  m_key_parts_suit; /* A number of key_parts suited to lookup
> > */
> >
> > that's very confusing name. Took me a while to understand what you meant.
> > Better call it m_usable_key_parts
> >
> > >bool master_had_triggers; /* set after tables opening */
> > >
> > > +  uint key_parts_suit_event(const KEY *key) const;
> >
> > and this could be "get_usable_key_parts()"
> 
> Okay, no problem. Both sound suitable, or usable, for me:)

and is similar to KEY::usable_key_parts

> > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> > > index 422d496d5c3..25705d13641 100644
> > > --- a/sql/log_event_server.cc
> > > +++ b/sql/log_event_server.cc
> > > @@ -6062,7 +6065,7 @@ int Rows_log_event::do_apply_event(rpl_group_info 
> > > *rgi)
> > >  */
> > >  sql_mode_t saved_sql_mode= thd->variables.sql_mode;
> > >  if (!is_auto_inc_in_extra_columns())
> > > -  thd->variables.sql_mode= (rpl_data.is_online_alter() ? 
> > > saved_sql_mode :0)
> > > +  thd->variables.sql_mode= (m_online_alter ? saved_sql_mode :0)
> > > | MODE_NO_AUTO_VALUE_ON_ZERO;
> >
> > wouldn't it be more correct to use saved_sql_mode also in normal
> > replication?
> 
> using MODE_NO_AUTO_VALUE_ON_ZERO was a fix to Bug #56662  Assertion failed:
> next_insert_id == 0, file .\handler.cc.
> See commit d5bf6b8aa8bb

No, I didn't mean to ask "why MODE_NO_AUTO_VALUE_ON_ZERO", I meant to
ask why not

  thd->variables.sql_mode= saved_sql_mode | MODE_NO_AUTO_VALUE_ON_ZERO;

that is why not "to use saved_sql_mode also in normal replication"

> > > +  uint p= 0;
> > > +  for (;p < key->ext_key_parts; p++)
> > > +  {
> > > +Field *f= key->key_part[p].field;
> > > +uint non_deterministic_default= f->default_value &&
> > > + f->default_value->flags | 
> > > VCOL_NOT_STRICTLY_DETERMINISTIC;
> > > +
> > > +int next_number_field= f->table->next_number_field ?
> > > +   f->table->next_number_field->field_index : -1;
> >
> > eh?
> > 1. why it's *in the loop* ?
> > 2. why not simply
> >
> >   || f == f->table->next_number_field
> >
> >or, to be extra safe
> >
> >   || f->table->field[f->field_index] == f->table->next_number_field
> >
> Thanks!
> Turns out there's no need in extra safety, it just works.

by "extra safety" I meant the case when

  f != f->table->field[f->field_index]

(here f= key->key_part[p].field). This is the case for varchar/blob
prefix keys. But as they cannot be auto-increment anyway, the direct
comparison should work, I suppose.

> > > +
> > > +if (f->field_index >= master_columns
> >
> > I don't think this condition ^^^ is needed. Same as elsewhere,
> > if f->field_index < master_columns but !bitmap_is_set,
> > does it mean the keyseg is ok? I don't think so.
> >
> It *is* needed.
> 
> An extra field with default can be used in the key lookup. A usual
> field should have an explicit value. If it's not, it means it's not
> sent (MINIMAL/NOBLOB) and we don't know what real value is

I don't understand how this answers my question. Let me rephrase.
What will break if you remove the condition

  f->field_index >= master_columns &&

?

> > > +&& !bitmap_is_set(_table->has_value_set, f->field_index)
> > > +&& (non_deterministic_default || next_number_field == 
> > > f->field_index))
> > > +  break;
> > > +  }
> > > +  return p;
> > > +}
> > >
> > >  /**
> > >Find the best key to us

Re: [Maria-developers] c83f87ece4a: MDEV-15990 REPLACE on a precise-versioned table returns duplicate key error (ER_DUP_ENTRY)

[Sergei Golubchik]

Hi, Nikita,

Sorry, this comes very late. But the bug is still present in 10.10,
and the filter sorts by the date precisely to highlight these old bugs.
So, here you are.

First comment, please, port it to the latest 10.3. I wasn't able to find
the repository with the fix, but the patch itself is avaliable from the
pull request as https://github.com/MariaDB/server/commit/c83f87ece4a.patch
so it can be salvaged.

More comments - see below.

On Oct 15, Nikita Malyavin wrote:
> From: Nikita Malyavin 
> Date: Fri, 20 Jul 2018 00:24:40 +1000
> Subject: [PATCH] MDEV-15990: REPLACE on a precise-versioned table returns
> * handle zero value in `row_start` `trx_id`
> * refuse to do more than one versioned insert on the same transaction and 
> same row
> * generalize versioned cases with and without triggers
> 
> diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc
> index dda469a1426d5..cd2690128de67 100644
> --- a/sql/sql_insert.cc
> +++ b/sql/sql_insert.cc
> @@ -1928,32 +1928,15 @@ int write_record(THD *thd, TABLE *table,COPY_INFO 
> *info)
>For system versioning wa also use path through delete since we 
> would
>save nothing through this cheating.
>  */
> -if (last_uniq_key(table,key_nr) &&
> +if (last_uniq_key(table,key_nr) && !table->versioned() &&

ok!

>  !table->file->referenced_by_foreign_key() &&
>  (!table->triggers || !table->triggers->has_delete_triggers()))
>  {
> -  if (table->versioned(VERS_TRX_ID) && table->vers_write)
> -  {
> -bitmap_set_bit(table->write_set, 
> table->vers_start_field()->field_index);
> -table->vers_start_field()->store(0, false);
> -  }
> -  if (unlikely(error= table->file->ha_update_row(table->record[1],
> - table->record[0])) 
> &&
> -  error != HA_ERR_RECORD_IS_THE_SAME)
> +  error= table->file->ha_update_row(table->record[1], 
> table->record[0]);
> +  if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME))
>  goto err;
>if (likely(!error))
> -  {
>  info->deleted++;
> -if (table->versioned(VERS_TIMESTAMP) && table->vers_write)
> -{
> -  store_record(table, record[2]);
> -  error= vers_insert_history_row(table);
> -  restore_record(table, record[2]);
> -  if (unlikely(error))
> -goto err;
> -}
> -  }
> -  else
>  error= 0;   // error was HA_ERR_RECORD_IS_THE_SAME
>
> thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row);
>/*
> @@ -1969,23 +1952,22 @@ int write_record(THD *thd, TABLE *table,COPY_INFO 
> *info)
>  TRG_ACTION_BEFORE, TRUE))
>  goto before_trg_err;
>  
> -  if (!table->versioned(VERS_TIMESTAMP))
> +  if (table->versioned(VERS_TRX_ID) && table->vers_write)
> +  {
> +bitmap_set_bit(table->write_set, 
> table->vers_start_field()->field_index);

this is conceptually wrong, you cannot set table->write_set in the middle
of statement execution. that is you can and it might even work for some
engines, but it breaks the API.

> +table->vers_start_field()->store(0, false);
> +  }
> +
> +  if (!table->versioned())
>  error= table->file->ha_delete_row(table->record[1]);
>else
> -  {
> -store_record(table, record[2]);
> -restore_record(table, record[1]);
> -table->vers_update_end();

why? is the row_end updated somewhere else?

>  error= table->file->ha_update_row(table->record[1],
>table->record[0]);
> -restore_record(table, record[2]);
> -  }
> -  if (unlikely(error))
> +  if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME))
>  goto err;
> -  if (!table->versioned(VERS_TIMESTAMP))
> +  if (likely(!error))
>  info->deleted++;
> -  else
> -info->updated++;
> +  error= 0;
>if (!table->file->has_transactions())
>  thd->transaction.stmt.modified_non_trans_table= TRUE;
>if (table->triggers &&
> @@ -1995,6 +1977,17 @@ int write_record(THD *thd, TABLE *table,COPY_INFO 
> *info)
>  trg_error= 1;
>  goto ok_or_after_trg_err;
>}
> +  if (table->versioned(VERS_TIMESTAMP) && table->vers_write)
> +  {
> +store_record(table, record[2]);
> +error = vers_insert_history_row(table);

if you do that, you shouldn't have been doing ha_update_row() above,
as far as I can see. What is that update above for?

> +restore_record(table, record[2]);
> +if 

Re: [Maria-developers] 19772f9c277: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations

[Sergei Golubchik]

Hi, Oleksandr,

On Oct 13, Oleksandr Byelkin wrote:
> revision-id: 19772f9c277 (mariadb-10.3.36-90-g19772f9c277)
> parent(s): e05ab0cfc5f
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-10-12 15:59:46 +0200
> message:
> 
> MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent 
> view operations
> 
> Prevent simultaniouse access to a view table SHARE (and so its MEM_ROOT) from 
> different threads.

If you're writing into share, why not to do it when a share is
populated, then you wouldn't need a mutex. And it'd be a logical place
to initialize the share anyway. Doing it at some arbitrary point in time
later looks strange.

I mean, why not to call this mariadb_view_version_get after the line

  share->view_def= sql_parse_prepare(, >mem_root, true);

where it logically belongs?

> +  mysql_mutex_lock(>LOCK_share);
>  
>if (!(share->tabledef_version.str=
>  (uchar*) alloc_root(>mem_root,
>  MICROSECOND_TIMESTAMP_BUFFER_SIZE)))

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b1b38b64598: MDEV-29181 Potential corruption on FK update on a table with vcol index

[Sergei Golubchik]

Hi, Marko,

On Oct 10, Marko Mäkelä wrote:
> On Sun, Oct 9, 2022 at 2:03 PM Sergei Golubchik  wrote:
> > I've spent more time on this issue, and it looks like there is no
> > structure in InnoDB with the life time till the end of the statement
> > (or, at least, nothing easily usable).
> 
> Would ha_innobase::m_prebuilt fill that purpose? Its scope is not
> exactly a statement, because ha_innobase::reset_template() will be
> invoked not only by ha_innobase::reset() at the end of a statement. As
> I can tell, it could be close enough.

prebuilt is only minimally reset, I thought you wouldn't want anything
complex there. And anyway, Nikita's problem is with cascading updates,
they don't have their own ha_innobase::m_prebuilt.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b1b38b64598: MDEV-29181 Potential corruption on FK update on a table with vcol index

[Sergei Golubchik]

Hi, Nikita,

On Sep 01, Sergei Golubchik wrote:
> 
> > This new solution simply does the following:
> > * Sets up a referenced table list in TABLE instance (sql_base.cc)
> > * Iterates through it along with dict_table_t::referenced_set
> >   (row_upd_check_references_constraints)
> > * Passes corresponding prebuilt through a call chain to
> >   row_ins_foreign_check_on_constraint
> > * Sets up newly created upd_node_t::prebuilt field and uses it
> > accordingly is cascade update.
> 
> Is this upd_node_t::prebuilt used anywhere? As a prebuilt, I mean.
> I couldn't find it (it's a complex patch, so I couldn've missed it).
> As far as I can see, it's only used to store a pointer to TABLE.
> 
> So it seems to me than a simpler fix for this bug could be:
> * remove vc_templ caching (mysql_table and mysql_table_query_id)
> * store TABLE* in upd_node_t.

I've spent more time on this issue, and it looks like there is no
structure in InnoDB with the life time till the end of the statement
(or, at least, nothing easily usable). So I suggest to keep the
mysql_table_query_id/mysql_table caching, but move it from dict_table_t
to upd_node_t. The life time will be till the end of the connection,
over many statements, that's why mysql_table_query_id will be still
needed.

Even better solution would be to put mysql_table_query_id/mysql_table
pair into ib_vcol_row and store ib_vcol_row in upd_node_t, instead of
creating it again for every row, saving on thousands of mallocs.
But I'm not sure it'll work for online alter in 10.11.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b1b38b64598: MDEV-29181 Potential corruption on FK update on a table with vcol index

[Sergei Golubchik]

Hi, Nikita,

On Sep 27, Nikita Malyavin wrote:
> Sergei, I have reworded the commit message, please see it here:
> https://github.com/MariaDB/server/commit/3a3064e355bac20ed56ae807e790068e16dd16f3

Same thing. I still cannot understand from the comment what the problem
was.

>> > This new solution simply does the following:
>> > * Sets up a referenced table list in TABLE instance (sql_base.cc)
>> > * Iterates through it along with dict_table_t::referenced_set
>> >   (row_upd_check_references_constraints)
>> > * Passes corresponding prebuilt through a call chain to
>> >   row_ins_foreign_check_on_constraint
>> > * Sets up newly created upd_node_t::prebuilt field and uses it
>> > accordingly is cascade update.
>>
>> Is this upd_node_t::prebuilt used anywhere? As a prebuilt, I mean.
>> I couldn't find it (it's a complex patch, so I couldn've missed it).
>> As far as I can see, it's only used to store a pointer to TABLE.
>>
>> So it seems to me than a simpler fix for this bug could be:
>> * remove vc_templ caching (mysql_table and mysql_table_query_id)
>> * store TABLE* in upd_node_t.
>>
> Not sure about the lifetimes, so it's not necessarily simpler.
> prebuilt is not used in this patch, but it is used during online row
> logging to convert the row to [my]sql format.

It is. But it's used as node->prebuilt->m_mysql_table.
And I write "store TABLE* in upd_node_t", so that'll work for online
alter too.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] dcfc60ea2ee: MDEV-29481 mariadb-upgrade prints confusing statement

[Sergei Golubchik]

Hi, Alexander,

See a couple of questions below:

On Oct 04, Alexander Barkov wrote:
> revision-id: dcfc60ea2ee (mariadb-10.4.26-28-gdcfc60ea2ee)
> parent(s): 6c2c825e503
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-10-03 10:32:42 +0400
> message:
> 
> MDEV-29481 mariadb-upgrade prints confusing statement
> 
> This is a new version of the patch instead of the reverted:
> 
>   MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade

> diff --git a/mysql-test/main/alter_table_debug_aria.test 
> b/mysql-test/main/alter_table_debug_aria.test
> new file mode 100644
> index 000..22df57686e1
> --- /dev/null
> +++ b/mysql-test/main/alter_table_debug_aria.test
> @@ -0,0 +1,36 @@
> +--source include/have_debug.inc
> +--source include/have_maria.inc
> +
> +--echo #
> +--echo # Start of 10.4 tests
> +--echo #
> +
> +--echo #
> +--echo # MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade
> +--echo #
> +
> +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA;
> +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=NOCOPY;
> +DROP TABLE t1;
> +
> +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA;
> +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=INSTANT;
> +DROP TABLE t1;
> +
> +SET SESSION debug_dbug='+d,emulate_pre_mdev_20704';
> +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA;

Why did you add these debug-only tests with the emulate_pre_mdev_20704 hack,
when you have perfect non-debug tests with real pre-MDEV-20704 created tables ?

> +SET SESSION debug_dbug='';
> +--error ER_ALTER_OPERATION_NOT_SUPPORTED
> +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=INSTANT;
> +DROP TABLE t1;
> +
> +SET SESSION debug_dbug='+d,emulate_pre_mdev_20704';
> +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA;
> +SET SESSION debug_dbug='';
> +--error ER_ALTER_OPERATION_NOT_SUPPORTED
> +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=NOCOPY;
> +DROP TABLE t1;
> +
> +--echo #
> +--echo # End of 10.4 tests
> +--echo #
> diff --git a/sql/sql_table.cc b/sql/sql_table.cc
> index 2436031200a..c192a4553a9 100644
> --- a/sql/sql_table.cc
> +++ b/sql/sql_table.cc
> @@ -6748,11 +6753,13 @@ Compare_keys compare_keys_but_name(const KEY 
> *table_key, const KEY *new_key,
> const KEY *const new_pk,
> const KEY *const old_pk)
>  {
> -  if (table_key->algorithm != new_key->algorithm)
> +  ulong key_flag_mask= HA_KEYFLAG_MASK |
> +   table->file->key_pack_flags_supported();

I don't understand. You said that these KEY->flags do not affect what the
engine does. Why to compare them at all?

> +
> +  if ((table_key->flags & key_flag_mask) != (new_key->flags & key_flag_mask))
>  return Compare_keys::NotEqual;
>  
> -  if ((table_key->flags & HA_KEYFLAG_MASK) !=
> -  (new_key->flags & HA_KEYFLAG_MASK))
> +  if (table_key->algorithm != new_key->algorithm)
>  return Compare_keys::NotEqual;
>  
>if (table_key->user_defined_key_parts != new_key->user_defined_key_parts)

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys

[Sergei Golubchik]

Hi, Nikita,

This review applies to the combined diff e2f8dff^..52f489e

On Oct 04, Nikita Malyavin wrote:

> diff --git a/sql/log_event.h b/sql/log_event.h
> index 91a910f..8452e665c98 100644
> --- a/sql/log_event.h
> +++ b/sql/log_event.h
> @@ -5275,8 +5278,11 @@ class Rows_log_event : public Log_event
>uchar*m_key;  /* Buffer to keep key value during searches */
>KEY  *m_key_info; /* Pointer to KEY info for m_key_nr */
>uint  m_key_nr;   /* Key number */
> +  uint  m_key_parts_suit; /* A number of key_parts suited to lookup */

that's very confusing name. Took me a while to understand what you meant.
Better call it m_usable_key_parts

>bool master_had_triggers; /* set after tables opening */
>  
> +  uint key_parts_suit_event(const KEY *key) const;

and this could be "get_usable_key_parts()"

> +
>int find_key(); // Find a best key to use in find_row()
>int find_row(rpl_group_info *);
>int write_row(rpl_group_info *, const bool);
> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> index 422d496d5c3..25705d13641 100644
> --- a/sql/log_event_server.cc
> +++ b/sql/log_event_server.cc
> @@ -6062,7 +6065,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi)
>  */
>  sql_mode_t saved_sql_mode= thd->variables.sql_mode;
>  if (!is_auto_inc_in_extra_columns())
> -  thd->variables.sql_mode= (rpl_data.is_online_alter() ? saved_sql_mode 
> :0)
> +  thd->variables.sql_mode= (m_online_alter ? saved_sql_mode :0)

wouldn't it be more correct to use saved_sql_mode also in normal replication?

> | MODE_NO_AUTO_VALUE_ON_ZERO;
>  
>  // row processing loop
> @@ -7949,14 +7952,35 @@ uint8 Write_rows_log_event::get_trg_event_map()
>  **/
>  
>  #if defined(HAVE_REPLICATION)
> -/*
> -  Compares table->record[0] and table->record[1]
> +/**
> +  @brief Compares table->record[0] and table->record[1]
> +
> +  @param masrter_columns  a number of columns on the source replica,
> +  0 if ONLINE ALTER TABLE
>  
> -  Returns TRUE if different.
> +  @returns true if different.
>  */
> -static bool record_compare(TABLE *table)
> +static bool record_compare(TABLE *table, uint master_columns)
>  {
> -  bool result= FALSE;
> +  bool result= false;
> +
> +  /*
> +Determine if the optimized check is possible (and we can
> +goto record_compare_exit).
> +We should ensure that all extra columns (i.e. fieldnr > master_columns)
> +have explicit values. If not, we will exclude them from comparison,
> +as they can contain non-deterministic values.
> +
> +master_columns == 0 case is optimization for ONLINE ALTER to check
> +all columns fast.
> +   */
> +  bool all_values_set= master_columns == 0
> +   && bitmap_is_set_all(>has_value_set);
> +  for (uint i = master_columns; all_values_set && i < table->s->fields; i++)
> +  {
> +all_values_set= bitmap_is_set(>has_value_set, i);
> +  }

wait, this doesn't make any sense. You only do the loop if
all_values_set is true.
But it is true only if bitmap_is_set_all(>has_value_set).
There is no point in doing bitmap_is_set() for individual bits,
when you already know that all bits are set :)

shouldn't it simply be

   bool all_values_set=  bitmap_is_set_all(>has_value_set);

?
but see below, perhaps you don't need all_values_set at all

> +
>/**
>  Compare full record only if:
>  - there are no blob fields (otherwise we would also need 
> @@ -7969,40 +7993,43 @@ static bool record_compare(TABLE *table)
>  */
>if ((table->s->blob_fields + 
> table->s->varchar_fields + 
> -   table->s->null_fields) == 0)
> +   table->s->null_fields) == 0
> +  && all_values_set)
>{
>  result= cmp_record(table,record[1]);
>  goto record_compare_exit;
>}
>  
>/* Compare null bits */
> -  if (memcmp(table->null_flags,
> -  table->null_flags+table->s->rec_buff_length,
> -  table->s->null_bytes))
> +  if (all_values_set && memcmp(table->null_flags,
> +   table->null_flags+table->s->rec_buff_length,
> +   table->s->null_bytes))
>{
> -result= TRUE;// Diff in NULL value
> +result= true;// Diff in NULL value
>  goto record_compare_exit;
>}
>  
>/* Compare fields */
>for (Field **ptr=table->field ; *ptr ; ptr++)
>{
> +Field *f= *ptr;
>  if (table->versioned() && (*ptr)->vers_sys_field())
>  {
>continue;
>  }
> -/**
> -  We only compare field contents that are not null.
> -  NULL fields (i.e., their null bits) were compared 
> -  earlier.
> -*/
> -if (!(*(ptr))->is_null())
> +/*
> +  We only compare fields that exist on the source (or in ONLINE
> +  ALTER case, that were in the original 

Re: [Maria-developers] 5265f7001c6: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared

[Sergei Golubchik]

Hi, Oleksandr,

On Sep 28, Oleksandr Byelkin wrote:
> > > +bool TABLE_LIST::is_the_same_definition(THD* thd, TABLE_SHARE *s)
> > > +{
> > > +  enum enum_table_ref_type tp= s->get_table_ref_type();
> > > +  if (m_table_ref_type == tp)
> > > +  {
> > > +bool res= m_table_ref_version == s->get_table_ref_version();
> > > +
> > > +/*
> > > +  If definition is different check content version
> > > +*/
> >
> > if res == true, why do you need to check tabledef_version?
> >
> 
> You are right, fixed:
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -8881,10 +8881,11 @@ bool TABLE_LIST::is_the_same_definition(THD* thd,
> TABLE_SHARE *s)
>  /*
>If definition is different check content version
>  */
> -if (tabledef_version.length &&
> -tabledef_version.length == s->tabledef_version.length &&
> -memcmp(tabledef_version.str, s->tabledef_version.str,
> -   tabledef_version.length) == 0)
> +if (res ||
> +(tabledef_version.length &&
> + tabledef_version.length == s->tabledef_version.length &&
> + memcmp(tabledef_version.str, s->tabledef_version.str,
> +tabledef_version.length) == 0))

I still don't understand. I'd expect something like

   if (!res &&
   (tabledef_version.length && ...

that is, if m_table_ref_version matches, then you don't need to do any
further checks, the table is fine. If m_table_ref_version differs,
meaning, the table was reopened, then you check tabledef_version and
triggers.

Perhaps the simplest fix would be to remove res and instead do

  if (m_table_ref_version == s->get_table_ref_version())
return TRUE;

  if (tabledef_version.length && ...

>  {
>if (table && table->triggers)
>{

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 5265f7001c6: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared

[Sergei Golubchik]

Hi, Oleksandr,

Few minor comments, see below

On Sep 27, Oleksandr Byelkin wrote:
> revision-id: 5265f7001c6 (mariadb-10.3.36-60-g5265f7001c6)
> parent(s): 86953d3df0a
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-09-27 10:23:31 +0200
> message:
> 
> MDEV-17124: mariadb 10.1.34, views and prepared statements:  ERROR 1615 
> (HY000): Prepared statement needs to be re-prepared
> 
> The problem is that if table definition cache (TDC) is full of real tables
> which are in tables cache, view definition can not stay there so will be
> removed by its own underlying tables.
> In situation above old mechanism of detection matching definition in PS
> and current version always require reprepare and so prevent executing
> the PS.
> 
> One work around is to increase TDC, other - improve version check for
> views/triggers (which is done here). Now in suspicious cases we check:
>  - timestamp (microseconds) of the view to be sure that version really
>have changed;
>  - time (microseconds) of creation of a trigger related to time
>(microseconds) of statement preparation.
 
> diff --git a/sql/table.cc b/sql/table.cc
> index 506195127b2..f289c2052cb 100644
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -8861,6 +8861,63 @@ bool TABLE_LIST::is_with_table()
>return derived && derived->with_element;
>  }
>  
> +
> +/**
> +  Check if the definition are the same.
> +
> +  If versions do not match it check definitions (with checking and setting
> +  trigger definition versions (times)
> +
> +  @sa check_and_update_table_version()
> +*/
> +
> +bool TABLE_LIST::is_the_same_definition(THD* thd, TABLE_SHARE *s)
> +{
> +  enum enum_table_ref_type tp= s->get_table_ref_type();
> +  if (m_table_ref_type == tp)
> +  {
> +bool res= m_table_ref_version == s->get_table_ref_version();
> +
> +/*
> +  If definition is different check content version
> +*/

if res == true, why do you need to check tabledef_version?

> +if (tabledef_version.length &&
> +tabledef_version.length == s->tabledef_version.length &&
> +memcmp(tabledef_version.str, s->tabledef_version.str,
> +   tabledef_version.length) == 0)
> +{
> +  if (table && table->triggers)
> +  {
> +my_hrtime_t hr_stmt_prepare= thd->hr_prepare_time;
> +if (hr_stmt_prepare.val)
> +  for(uint i= 0; i < TRG_EVENT_MAX; i++)
> +for (uint j= 0; j < TRG_ACTION_MAX; j++)
> +{
> +  Trigger *tr=
> +table->triggers->get_trigger((trg_event_type)i,
> + (trg_action_time_type)j);
> +  if (tr)
> +if (hr_stmt_prepare.val <= tr->hr_create_time.val)
> +{
> +  set_tabledef_version(s);
> +  return FALSE;
> +}
> +}
> +  }
> +  set_table_id(s);
> +  return TRUE;
> +}
> +else
> +  tabledef_version.length= 0;
> +return res;
> +  }
> +  else
> +set_tabledef_version(s);

why not set_table_id() ?

> +  return FALSE;
> +}
> +
> +
>  uint TABLE_SHARE::actual_n_key_parts(THD *thd)
>  {
>return use_ext_keys &&
> diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h
> index ae3d1738b16..f7a2ccf2abc 100644
> --- a/sql/sql_trigger.h
> +++ b/sql/sql_trigger.h
> @@ -198,7 +198,7 @@ class Table_triggers_list: public Sql_alloc
>*/
>List definition_modes_list;
>/** Create times for triggers */
> -  List create_times;
> +  List hr_create_times;

I suspect it might be UB. You use FILE_OPTIONS_ULLLIST for hr_create_times,
perhaps it's safer to use List here.

>  
>List  definers_list;
>  
> @@ -328,4 +328,14 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST 
> *tables, bool create);
>  extern const char * const TRG_EXT;
>  extern const char * const TRN_EXT;
>  
> +
> +/**
> +  Make time compatible with MySQL 5.7 trigger time.
> +*/
> +
> +inline my_hrtime_t make_hr_time(my_time_t time, ulong time_sec_part)
> +{
> +  return my_hrtime_t({((ulonglong) time)*100 + time_sec_part});
> +}

better put it next to hrtime_to_time/etc
and, please, always 'static inline'

> +
>  #endif /* SQL_TRIGGER_INCLUDED */
> diff --git a/sql/sql_view.cc b/sql/sql_view.cc
> index 17dea643144..177d01a312e 100644
> --- a/sql/sql_view.cc
> +++ b/sql/sql_view.cc
> @@ -1154,7 +1163,32 @@ static int mysql_register_view(THD *thd, TABLE_LIST 
> *view,
>DBUG_RETURN(error);
>  }
>  
> +/**
> +  Check is TABLE_LEST and SHARE match
> +  @param[in]  viewTABLE_LIST of the view
> +  @param[in]  share   Share object of view
> +
> +  @return false on error or misspatch

Eh. I don't understad this comment at all. First, I thought you made two
typos, s/is/if/ and s/misspatch/mismatch/. But this function doesn't check if
something matches, if gets the view version, as the name says, the name's
good. But the comment seems to be from is_the_same_definition()

> +*/
> +

Re: [Maria-developers] 349283c5e7a: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared

[Sergei Golubchik]

Hi, Oleksandr,

On Sep 26, Oleksandr Byelkin wrote:
> >
> > > > > +timestamp.second_part= trigger->ms_create_time % 100;
> > > >
> > > > hrtime_to_time() and hrtime_sec_part()
> > >
> > > they made for events, so I made hr_time_to_time, hr_time_from_time
> >
> > What do you mean by "made for events" ?
> 
> It request val field from passed object (I mean the macro), and I am
> not sure if rewriting that macro a good idea.

it works with my_hrtime_t values, so if you store high-resolution time
values in my_hrtime_t than these macros will work :)

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 349283c5e7a: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared

[Sergei Golubchik]

Hi, Oleksandr,

On Sep 26, Oleksandr Byelkin wrote:
> > >LEX_STRING name; /* name for named prepared statements */
> > >LEX *lex; // parse tree descriptor
> > > +  ulonglong ms_prepare_time; // time of preparation in microseconds
> >
> > "ms" here too :(
> > Better just say
> >
> > my_hrtime_t prepare_time;
> 
> I changed everywhere to hr_

Yes, but you kept them ulonglong and not my_hrtime_t, why?
my_hrtime_t is a type for storing high resolution time values.

> > > +timestamp.second_part= trigger->ms_create_time % 100;
> >
> > hrtime_to_time() and hrtime_sec_part()
> 
> they made for events, so I made hr_time_to_time, hr_time_from_time

What do you mean by "made for events" ?

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] f5a8327d016: MDEV-29458: Role grant commands do not propagate all grants

[Sergei Golubchik]

Hi, Vicentiu,

On Sep 12, Vicentiu Ciorbaru wrote:
> revision-id: f5a8327d016 (mariadb-10.3.36-26-gf5a8327d016)
> parent(s): fa75057b238
> author: Vicențiu Ciorbaru
> committer: Vicențiu Ciorbaru
> timestamp: 2022-09-05 16:40:10 +0300
> message:
> 
> MDEV-29458: Role grant commands do not propagate all grants
> 
> diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc
> index ba457083b75..3806fb31aa6 100644
> --- a/sql/sql_acl.cc
> +++ b/sql/sql_acl.cc
> @@ -2741,7 +2741,9 @@ static void acl_update_role(const char *rolename, ulong 
> privileges)
>  {
>ACL_ROLE *role= find_acl_role(rolename);
>if (role)
> +  {
>  role->initial_role_access= role->access= privileges;
> +  }
>  }

I wish you put this and the tab-to-spaces change in a separate commit.
(even better - tab-to-spaces in a separate commit and this hunk - nowhere :)

> @@ -9720,6 +9725,14 @@ static int handle_grant_struct(enum enum_acl_lists 
> struct_no, bool drop,
>  remove_ptr_from_dynarray(>role_grants, acl_role);
>}
>  
> +  /* Remove all of the role_grants from this user. */
> +  while (acl_role->role_grants.elements)
> +delete_dynamic_element(_role->role_grants,
> +   acl_role->role_grants.elements - 1);

this is a weird way of deleting all elements in the array. What's wrong
with

delete_dynamic(_role->role_grants);

?

> +
> +  /* all grants must be revoked from this role by now. propagate this */
> +  propagate_role_grants(acl_role, PRIVS_TO_MERGE::ALL);
> +
>my_hash_delete(_roles, (uchar*) acl_role);
>DBUG_RETURN(1);
>  }
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] e0e75de899f: MDEV-26715 Windows/installer - allow passwordless login for root

[Sergei Golubchik]

Hi, Vladislav,

Note, despite what the subject says, below is not for one commit, but
for the diff over them all.

Just a couple of questions and a comment:

> diff --git a/libmariadb b/libmariadb
> index a6665e65c05..22f7ed12b46 16
> --- a/libmariadb
> +++ b/libmariadb
> @@ -1 +1 @@
> -Subproject commit a6665e65c05c00f9df983a0f788a643ca7bc084c
> +Subproject commit 22f7ed12b4645a9fc5d66974551359373090c86b

are auth_gssapi plugins using only standard Windows libraries?

> diff --git a/mysql-test/main/mysql_install_db_win_utf8.test 
> b/mysql-test/main/mysql_install_db_win_utf8.test
> index fc67b66d3ca..0dc75af3914 100644
> --- a/mysql-test/main/mysql_install_db_win_utf8.test
> +++ b/mysql-test/main/mysql_install_db_win_utf8.test
> @@ -1,6 +1,6 @@
>  --source include/windows.inc
>  --source include/check_utf8_cli.inc
> -
> +use mysql;

Why?

>  # Create database  in tmp directory using mysql_install_db.exe,
>  # and start server from this directory.
>  let $ddir= $MYSQLTEST_VARDIR/tmp/датадир;
> diff --git a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm 
> b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm
> index e77ba05cb5c..71a6bf33b7b 100644
> --- a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm
> +++ b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm
> @@ -1,8 +1,9 @@
> +
>  package My::Suite::AuthGSSAPI;
>  
>  @ISA = qw(My::Suite);
>  
> -return "No AUTH_GSSAPI plugin" unless $ENV{AUTH_GSSAPI_SO};
> +return "No AUTH_GSSAPI plugin" unless ($ENV{AUTH_GSSAPI_SO} or ($^O eq 
> 'MSWin32'));

The second condition normally is something like

  or $::mysqld_variables{gssapi} eq "ON";

>  
>  return "Not run for embedded server" if $::opt_embedded_server;
>  

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 3f9e6c9c111: MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds)

[Sergei Golubchik]

Hi, Oleksandr,

On Sep 09, Oleksandr Byelkin wrote:
> revision-id: 3f9e6c9c111 (mariadb-10.3.35-36-g3f9e6c9c111)
> parent(s): 5feb60ce186
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-06-24 15:37:25 +0200
> message:
> 
> MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in 
> handle_grant_table instead of ERROR | Buffer overflow (on optimized builds)
> 
> Return an error if we can not read a table required for ACL.
> 
> diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc
> index 4c68b4505a4..38d2e9c35a1 100644
> --- a/sql/sql_acl.cc
> +++ b/sql/sql_acl.cc
> @@ -7420,6 +7420,11 @@ static bool grant_load(THD *thd,
>while (!p_table->file->ha_index_next(p_table->record[0]));
>  }
>}
> +  else
> +  {
> +sql_print_error("Missing system table mysql.procs_priv; "
> +"please run mysql_upgrade to create it");
> +  }

I don't think so. The comment earlier says

p_table= procs_priv.table(); // this can be NULL

and if "this can be NULL" then it's not an error if it is.

>  
>  end_unlock_p:
>if (p_table)
> @@ -10041,6 +10046,8 @@ static int handle_grant_data(THD *thd, Grant_tables& 
> tables, bool drop,
>}
>  
>/* Handle stored routines table. */
> +  if (tables.procs_priv_table().table_exists())
> +  {

This is ok, but also, please, check that all accesses to other optional
tables are properly protected. See FIRST_OPTIONAL_TABLE define, all
tables below it may not exist.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 68ebe0e2f5e: MDEV-29446 Change SHOW CREATE TABLE to display default collation

[Sergei Golubchik]

Hi, Alexander,

On Sep 09, Alexander Barkov wrote:
> revision-id: 68ebe0e2f5e (mariadb-10.3.36-25-g68ebe0e2f5e)
> parent(s): f501f815bc7
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-09-07 08:10:58 +0400
> message:
> 
> MDEV-29446 Change SHOW CREATE TABLE to display default collation
> 
> diff --git a/sql/sql_show.cc b/sql/sql_show.cc
> index a019c3c9dbd..cdf0bc01b6a 100644
> --- a/sql/sql_show.cc
> +++ b/sql/sql_show.cc
> @@ -2253,18 +2247,6 @@ int show_create_table(THD *thd, TABLE_LIST 
> *table_list, String *packet,
>{
>  packet->append(STRING_WITH_LEN(" CHARACTER SET "));
>  packet->append(field->charset()->csname);
> -  }
> -  /*
> -For string types dump collation name only if
> -collation is not primary for the given charset
> -
> -For generated fields don't print the COLLATE clause if
> -the collation matches the expression's collation.
> -  */
> -  if (!(field->charset()->state & MY_CS_PRIMARY) &&
> -  (!field->vcol_info ||
> -   field->charset() != field->vcol_info->expr->collation.collation))

why, I think, this second condition could still apply, no need to remove
it.

> -  {
>  packet->append(STRING_WITH_LEN(" COLLATE "));
>  packet->append(field->charset()->name);
>}

Note that in 10.10 you need to fix json_table code
inside a big if() condition:

  ...
  (!(m_explicit_cs->state & MY_CS_PRIMARY) &&
   (str->append(STRING_WITH_LEN(" COLLATE ")) ||
str->append(_explicit_cs->coll_name) ||
  ...

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 489a7fba324: MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event

[Sergei Golubchik]

Hi, Andrei,

I still don't understand this.

1. Why rgi->options_to_bin_log is only set for GTID_EVENT?
   Is there a guarantee that there always be a GTID_EVENT before a
   QUERY_EVENT? What if gtids aren't enabled?

2. how do you guarantee that all query events for a previous value of
   options_written_to_bin_log have been applied before updating it
   in the rgi?

3. and if you do guarantee it, why is it atomic? No query events =
   nobody reads options_written_to_bin_log concurrently.

And why not to do a simple patch, like the one I've attached?
(note, it's an intentionally minimal patch to show the fix, practically
I'd accompany it with a small cleanup patch)

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

On Sep 02, Andrei Elkin wrote:
> 
> > Query_log_event cannot just randomly use some
> > options_written_to_bin_log, it must use options_written_to_bin_log
> > from the Format_description_log_event that came from the binlog file
> > where the Query_log_event was read from.
> 
> [ Not to engage you into a separate discussion,
>   still why 'randomly'? In the quoted description p.1 says
>   the Query_log_event uses `options_written_to_bin_log` from the
>   immediately preceding-in-binlog FD. ]
> 
> > If rli->relay_log.description_event_for_exec is not that
> > Format_description_log_event, and assuming that the correct
> > Format_description_log_event is not stored anywhere, the Query_log_event
> > has to remember options_written_to_bin_log itself,
> 
> It's an option, but the chosen one avoids such remembering per each
> Query event. Instead ...
> 
> > there's not much we can do about it.
> >
> > Or it can apply the mask in the constructor and store not flags2 but
> > the new value of thd->variables.option_bits.
> >
> > Or it can extend flags2 to cover all OPTIONS_WRITTEN_TO_BINLOG.
> 
> ... of taking care of individual events the patch basically goes along
> the base to use a placeholder that
> A. lasts (unlike FD instance) the whole slave session time, and
> B. the stored `options_written_to_binlog` is guaranteed to be a copy
>of the most recent FD before the Query
> 
> To help you to understand the method, here is a sequence diagram,
> where on the top line
> the slave SQL thread, its Relay-log-info object, a similar to that Worker's 
> instance,
> and a Worker thread.
> 
> 
> SQL thread   RLI/RGIWorker RGI Worker 
> |  ...
> --->+   read  FD   ..i
> |  ...
> +-->
>   stores options
> .
> --->+  read Query   
> |   .
> +-->+
>finds Worker RGI
>to store in theree a copy FD.options
> | .
> +>+
>schedules Query Worker |
> . | 
> .<+
>reads options  | 
>   |  applies
>  ... Query
> 
> It must be clear the A holds.
> To prove B let after Query a second FD^2 is read by the SQL thread and
> that FD^2 is from a different version server.
> Then the SQL thread always have a wait-for-workers-to-become-idle
> condition, so execution would go like this:
> 
> --->+  read Query   .... 
> |   . |
> +-->+ |
>finds Worker RGI   |
>to store in theree a copy FD.options   |
> | |
> --->+  read FD^2  |  applying of
> | |  Query
>...|
>... wait for Worker to complete Query  |
>...|
> +<|
> |
> +-->.
> | stores FD^2
> |   options
>...
> 
> So FD^2 can't affect queries from earlier binlogs.
> 
> I am all ears to hear about your feedback.
> 
> Cheers,
> 
> Andrei
diff --git a/sql/log_event.cc b/sql/log_event.cc
index 0aeec20b2ea..7cf27438700 100644
--- a/sql/log_event.cc
+++ b/sql/log_event.cc
@@ -1509,7 +1509,7 @@ Query_log_event::Query_log_event(const uchar *buf, uint 
event_len,
 switch (*pos++) {
 case Q_FLAGS2_CODE:
   CHECK_SPACE(pos, end, 4);
-  flags2_inited= 1;
+  flags2_inited= description_event->options_written_to_bin_log;
   flags2= uint4korr(pos);
   DBUG_PRINT("info",("In Query_log_event, read flags2: %lu", (ulong) 
flags2));
 

Re: [Maria-developers] 489a7fba324: MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event

[Sergei Golubchik]

Hi, Andrei,

On Sep 01, Andrei wrote:
> revision-id: 489a7fba324 (mariadb-10.5.17-10-g489a7fba324)
> parent(s): 0324bde8469
> author: Andrei
> committer: Andrei
> timestamp: 2022-09-01 14:15:11 +0300
> message:
> 
> MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event
> 
> The ASAN report was reported in the parallel slave execution of a Query
> event and implicitly involved (so also parallelly run) Format-Description
> event.
> The Query actually had dependency on a preceding "old" FD whose
> instance got destructed, so to cause the ASAN error.
> 
> The case is fixed with copying of the value of
> `options_written_to_bin_log` the sensitive member
> of FD event into worker's stable placeholders, so FD change
> won't affect Queries dependent on "old" instances.
> 
> The method observes the following properties:
> 0. each worker gets its private copy;
> 1. the copied value gets a part of the execution context for any
>Queries that follow the FD in binlog;
> 2. any new FD forces a new copy which does not any affect on
>the past (in binlog order) Queries which might be still in
>execution.

Sorry, but this doesn't make any sense. The value of
options_written_to_bin_log depends on the version of the server that has
written that binlog file. Query_log_event cannot just randomly use some
options_written_to_bin_log, it must use options_written_to_bin_log from
the Format_description_log_event that came from the binlog file where
the Query_log_event was read from.

If rli->relay_log.description_event_for_exec is not that
Format_description_log_event, and assuming that the correct
Format_description_log_event is not stored anywhere, the Query_log_event
has to remember options_written_to_bin_log itself, there's not much we
can do about it.

Or it can apply the mask in the constructor and store not flags2 but
the new value of thd->variables.option_bits.

Or it can extend flags2 to cover all OPTIONS_WRITTEN_TO_BINLOG.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b1b38b64598: MDEV-29181 Potential corruption on FK update on a table with vcol index

[Sergei Golubchik]

Hi, Nikita,

On Aug 30, Nikita Malyavin wrote:
> revision-id: b1b38b64598 (mariadb-10.5.17-4-gb1b38b64598)
> parent(s): 3b656ac8c17
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2022-08-23 00:27:39 +0300
> message:
> 
> MDEV-29181 Potential corruption on FK update on a table with vcol
> index
> 
> vc_templ->mysql_table concept is completely broken. This table pointer
> persists with a global access and can be overwritten by arbitrary
> thread. Like it wasn't enough, it also could become invalid after
> eviction from tc cache.

this is a bit misleading. I agree it's completely broken, though :)

But not "this table pointer", because, of course it persists with a
global access and can be overwritten and so on. This is by design, it's
protected by mysql_table_query_id.

The problem is that table (dict_table_t) argument of
innodb_find_table_for_vc() points to a shared data structure, not thread
local, as I believed. So vc_templ is shared too.

I suggest to rewrite the comment like

  vc_templ->mysql_table concept is completely broken.
  table argument of innodb_find_table_for_vc() points to a shared data
  structure dict_table_t, and thus vc_templ is shared too and cannot
  be used for a thread-local cache.

> This new solution simply does the following:
> * Sets up a referenced table list in TABLE instance (sql_base.cc)
> * Iterates through it along with dict_table_t::referenced_set
>   (row_upd_check_references_constraints)
> * Passes corresponding prebuilt through a call chain to
>   row_ins_foreign_check_on_constraint
> * Sets up newly created upd_node_t::prebuilt field and uses it
> accordingly is cascade update.

Is this upd_node_t::prebuilt used anywhere? As a prebuilt, I mean.
I couldn't find it (it's a complex patch, so I couldn've missed it).
As far as I can see, it's only used to store a pointer to TABLE.

So it seems to me than a simpler fix for this bug could be:
* remove vc_templ caching (mysql_table and mysql_table_query_id)
* store TABLE* in upd_node_t.
 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ed477a6d30c: MDEV-29056 Replica thread reports error on ALTER ONLINE after LOCK WRITE

[Sergei Golubchik]

Hi, Nikita,

No, in this case (and commonly for collateral changes) I'd rather prefer
to refine the original (878a92e) commit to not cause the MDEV-29056 in
the first place.

Ignoring LOCK= clause is logically correct and that's what the old code
was doing anyway. But perhaps ALTER TABLE still needs to check that
ALGORITHM= and LOCK= clauses are compatible?

On Aug 27, Nikita Malyavin wrote:
> revision-id: ed477a6d30c (mariadb-10.6.1-521-ged477a6d30c)
> parent(s): 56206e60315
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2022-07-26 00:41:00 +0300
> message:
> 
> MDEV-29056 Replica thread reports error on ALTER ONLINE after LOCK WRITE
> 
> Commit 878a92e (MDEV-28943) changes the behavior of ALTER TABLE under
> prelocking. It now ignores passed LOCK= value in that case.
> 
> However, LOCK TABLES command is never replicated, so the replica node
> remains unaware of it.
> 
> The solution would be to guess on the replica side on the mode used by
> master. To make a deduction reliable, master's locked_tables_mode state
> is passed for query log events.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] e2f8dffc056: MDEV-29069 ER_KEY_NOT_FOUND on online autoinc addition + concurrent DELETE

[Sergei Golubchik]

Hi, Nikita,

Few comments below

On Aug 27, Nikita Malyavin wrote:
> commit e2f8dffc056
> Author: Nikita Malyavin 
> Date:   Mon Jul 25 23:52:24 2022 +0300
> 
> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> index 422d496d5c3..ca17c5c07a8 100644
> --- a/sql/log_event_server.cc
> +++ b/sql/log_event_server.cc
> @@ -7949,14 +7952,35 @@ uint8 Write_rows_log_event::get_trg_event_map()
>  **/
>  
>  #if defined(HAVE_REPLICATION)
> -/*
> -  Compares table->record[0] and table->record[1]
> +/**
> +  @brief Compares table->record[0] and table->record[1]
> +
> +  @param masrter_columns  a number of columns on the source replica,
> +  0 if ONLINE ALTER TABLE
>  
> -  Returns TRUE if different.
> +  @returns true if different.
>  */
> -static bool record_compare(TABLE *table)
> +static bool record_compare(TABLE *table, uint master_columns)
>  {
> -  bool result= FALSE;
> +  bool result= false;
> +
> +  /*
> +Determine if the optimized check is possible (and we can
> +goto record_compare_exit).
> +We should ensure that all extra columns (i.e. fieldnr > master_columns)
> +have explicit values. If not, we will exclude them from comparison,
> +as they can contain non-deterministic values.
> +
> +master_columns == 0 case is optimization for ONLINE ALTER to check
> +all columns fast.
> +   */
> +  bool all_values_set= master_columns == 0
> +   && bitmap_is_set_all(>has_value_set);
> +  for (uint i = master_columns; all_values_set && i < table->s->fields; i++)
> +  {
> +all_values_set= bitmap_is_set(>has_value_set, i);
> +  }

it'd be safer and faster to do

 all_values_set= bitmap_is_set_all(>has_value_set);

and, please, add replication (not online alter) tests:
1. slave has more columns
2. binlog row format is "minimal", this might trigger it too

> +
>/**
>  Compare full record only if:
>  - there are no blob fields (otherwise we would also need 
> @@ -7969,47 +7993,68 @@ static bool record_compare(TABLE *table)
>  */
>if ((table->s->blob_fields + 
> table->s->varchar_fields + 
> -   table->s->null_fields) == 0)
> +   table->s->null_fields) == 0
> +  && all_values_set)
>{
>  result= cmp_record(table,record[1]);
>  goto record_compare_exit;
>}
>  
>/* Compare null bits */
> -  if (memcmp(table->null_flags,
> +  if (all_values_set && memcmp(table->null_flags,
> table->null_flags+table->s->rec_buff_length,
> table->s->null_bytes))
>{
> -result= TRUE;// Diff in NULL value
> +result= true;// Diff in NULL value
>  goto record_compare_exit;
>}
>  
>/* Compare fields */
>for (Field **ptr=table->field ; *ptr ; ptr++)
>{
> +Field *f= *ptr;
>  if (table->versioned() && (*ptr)->vers_sys_field())
>  {
>continue;
>  }
> -/**
> -  We only compare field contents that are not null.
> -  NULL fields (i.e., their null bits) were compared 
> -  earlier.
> +/*
> +  We only compare fields that exist on the source (or in ONLINE
> +  ALTER case, that were in the original table). For reference,
> +  replica tables can also contain extra fields.
>   */
> -if (!(*(ptr))->is_null())
> -{
> -  if ((*ptr)->cmp_binary_offset(table->s->rec_buff_length))
> +if (f->field_index > master_columns && !f->has_explicit_value())
> +  continue;

why do you check for f->field_index > master_columns?
isn't !f->has_explicit_value() the one that decided?
in what case will be ok to compare values when
f->field_index < master_columns but no explicit value was set?

> +
> +if (f->is_null() != f->is_null(table->s->rec_buff_length)
> +|| f->cmp_binary_offset(table->s->rec_buff_length))
>  {
> -result= TRUE;
> +  result= true;
>goto record_compare_exit;
>  }
>}
> -  }
>  
>  record_compare_exit:
>return result;
>  }
>  
> +/**
> +  Newly added fields with non-deterministic defaults (i.e. DEFAULT(RANDOM()),
> +  CURRENT_TIMESTAMP, AUTO_INCREMENT) should be excluded from key search.
> +  Basically we exclude all the default-filled fields based on
> +  has_explicit_value bitmap.
> + */
> +bool Rows_log_event::key_suits_event(const KEY *key) const
> +{
> +  uint master_columns= m_online_alter ? 0 : m_cols.n_bits;
> +  for (uint p= 0; p < key->ext_key_parts; p++)
> +  {
> +uint field_idx= key->key_part[p].fieldnr - 1;
> +if (field_idx >= master_columns
> +&& !bitmap_is_set(_table->has_value_set, field_idx))
> +  return false;
> +  }
> +  return true;
> +}
>  
>  /**
>Find the best key to use when locating the row in @c find_row().
> @@ -8030,6 +8075,18 @@ int Rows_log_event::find_key()
>DBUG_ENTER("Rows_log_event::find_key");
>

Re: [Maria-developers] 5154e224a97: MDEV-5215 Granted to PUBLIC

[Sergei Golubchik]

Hi, Oleksandr,

> commit 5154e224a97
> Author: Oleksandr Byelkin 
> Date:   Mon Dec 13 16:15:21 2021 +0100
> 
> diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql
> index fd2f1c95dda..a1721913b2c 100644
> --- a/scripts/mysql_system_tables.sql
> +++ b/scripts/mysql_system_tables.sql
> @@ -47,6 +47,10 @@ INSERT IGNORE INTO global_priv SELECT * FROM tmp_user_sys 
> WHERE 0 <> @need_sys_u
>  DROP TABLE tmp_user_sys;
>  
>  
> +-- This special "role" needed for GRAND ... TO PUBLIC
> +INSERT IGNORE INTO global_priv (Host,User,Priv) VALUES ('', 'PUBLIC', 
> concat('{"access":0,"version_id":',regexp_replace(regexp_replace(@@version, 
> '\\b\\d\\b', '0\\0'), '\\D', ''),',"is_role":true}'));

1. why here and not in mysql_system_tables_data.sql ?
2. why this version magic? mysql_system_tables_data.sql doesn't do it for other 
accounts
3. why do you do it at all? I thought the server can create this role as needed

> +
> +
>  CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT 
> EXISTS user AS SELECT
>Host,
>User,
> @@ -95,7 +99,9 @@ CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY 
> DEFINER VIEW IF NOT EXISTS
>ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role,
>IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role,
>CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS 
> DECIMAL(12,6)) AS max_statement_time
> -  FROM global_priv;
> +  FROM global_priv
> +-- Do not show special role for GRANT TO PUBLIC
> +  WHERE not (Host = "" and User = "PUBLIC");

Why? I don't think it's wrong to show a new PUBLIC role.

>  
>  -- Remember for later if user table already existed
>  set @had_user_table= @@warning_count != 0;
> diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc
> index 601cae2e945..aa9bd91ec52 100644
> --- a/sql/sql_acl.cc
> +++ b/sql/sql_acl.cc
> @@ -316,6 +318,13 @@ static bool show_table_and_column_privileges(THD *, 
> const char *, const char *,
>  static int show_routine_grants(THD *, const char *, const char *,
> const Sp_handler *sph, char *, int);
>  
> +static ACL_ROLE *acl_public= NULL;
> +
> +inline privilege_t public_access()
> +{
> +  return (acl_public ? acl_public->access : NO_ACL);
> +}

Why not to have acl_public always not null?
e.g. define it as

  static ACL_ROLE acl_public;

so it's not even a pointer. And don't store it in the dynamic array.

Or make it like

  static ACL_ROLE acl_public_dummy, *acl_public= _public_dummy;

then you can keep he real role in the dynamic array, but still
acl_public will never be null.

but in the standard PUBLIC is not a role at all, so may be it'd be
simpler not to store it in the dynamic array

> +
>  class Grant_tables;
>  class User_table;
>  class Proxies_priv_table;
> @@ -3148,7 +3183,7 @@ bool acl_getroot(Security_context *sctx, const char 
> *user, const char *host,
>}
>else // Role, not User
>{
> -ACL_ROLE *acl_role= find_acl_role(user);
> +ACL_ROLE *acl_role= find_acl_role(user, false);

are there tests for that?

>  if (acl_role)
>  {
>res= 0;
> @@ -3161,6 +3196,26 @@ bool acl_getroot(Security_context *sctx, const char 
> *user, const char *host,
>  }
>}
>  
> +  /*
> +PUBLIC magic:
> +
> +Note: for usual user privileges of 3 component merged together:
> +  1) user privileges
> +  2) set role privileges
> +  3) public privileges
> +But for this routine (used in Security_context::change_security_context)
> +only 2 component merged:
> +  1) user OR role privileges we are switching to

because if we're switching to a user, there can be no set role?
I feel this comment is more confusing, then helping, really,
I'd suggest to remove it completely

> +  2) public privileges
> +  */
> +  if (acl_public)
> +  {
> +if (ACL_DB *acl_db= acl_db_find(db, public_name.str, "", "", FALSE))
> +  sctx->db_access|= acl_db->access;
> +
> +sctx->master_access|= acl_public->access;
> +  }
> +
>mysql_mutex_unlock(_cache->lock);
>DBUG_RETURN(res);
>  }
> @@ -3320,20 +3375,22 @@ int acl_setrole(THD *thd, const char *rolename, 
> privilege_t access)
>/* merge the privileges */
>Security_context *sctx= thd->security_ctx;
>sctx->master_access= access;
> -  if (thd->db.str)
> -sctx->db_access= acl_get(sctx->host, sctx->ip, sctx->user, thd->db.str, 
> FALSE);
> -
>if (!strcasecmp(rolename, "NONE"))
>{
>  thd->security_ctx->priv_role[0]= 0;
>}
>else
>{
> -if (thd->db.str)
> -  sctx->db_access|= acl_get("", "", rolename, thd->db.str, FALSE);
>  /* mark the current role */
>  strmake_buf(thd->security_ctx->priv_role, rolename);
>}
> +  if (thd->db.str)
> +sctx->db_access= acl_get_all3(sctx, thd->db.str, FALSE);
> +
> +  // PUBLIC magic
> +  if (acl_public)
> +sctx->master_access|= acl_public->access;

why not to do it in check_user_can_set_role()?
Like

-   *access = 

Re: [Maria-developers] bcfc424d859: MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite

[Sergei Golubchik]

Hi, Oleksandr,

On Aug 25, Oleksandr Byelkin wrote:
> revision-id: bcfc424d859 (mariadb-10.6.1-483-gbcfc424d859)
> parent(s): 1c3182a99e5
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-08-16 11:49:41 +0200
> message:
> 
> MDEV-5215 prerequisite: remove test and test_* database hacks in the test 
> suite

let's say, it's ok.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 1c3182a99e5: MDEV-5215 prerequisite of prerequisite: if DB is not mentioned in connect ignore errors of switching to it

[Sergei Golubchik]

Hi, Oleksandr,

On Aug 25, Oleksandr Byelkin wrote:
> revision-id: 1c3182a99e5 (mariadb-10.6.1-482-g1c3182a99e5)
> parent(s): 7253cdf8928
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-08-16 11:49:41 +0200
> message:
> 
> MDEV-5215 prerequisite of prerequisite: if DB is not mentioned in connect 
> ignore errors of switching to it
> 
> diff --git a/mysql-test/main/init_connect.test 
> b/mysql-test/main/init_connect.test
> index e96d02fe0d1..60549c3458e 100644
> --- a/mysql-test/main/init_connect.test
> +++ b/mysql-test/main/init_connect.test
> @@ -10,31 +10,31 @@
>  
>  --source include/add_anonymous_users.inc
>  
> -connect (con0,localhost,root,,);
> +connect (con0,localhost,root,,test);

why did you specify a db explicitly (here and in many other places)?
I thought the whole point was to modify mysqltest in a way that would
avoid the need of changing tests.

>  connection con0;
>  select hex(@a);

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] b3844107287: MDEV-16546 System versioning setting to allow history modification

[Sergei Golubchik]

Hi, Aleksey,

On Aug 03, Aleksey Midenkov wrote:
> > > diff --git a/sql/log_event.h b/sql/log_event.h
> > > index 3adc7a26d93..dc269955c5f 100644
> > > --- a/sql/log_event.h
> > > +++ b/sql/log_event.h
> > > @@ -535,16 +535,12 @@ class String;
> > >  */
> > >  #define OPTIONS_WRITTEN_TO_BIN_LOG \
> > >(OPTION_AUTO_IS_NULL | OPTION_NO_FOREIGN_KEY_CHECKS |  \
> > > -   OPTION_RELAXED_UNIQUE_CHECKS | OPTION_NOT_AUTOCOMMIT | 
> > > OPTION_IF_EXISTS)
> > > +   OPTION_RELAXED_UNIQUE_CHECKS | OPTION_NOT_AUTOCOMMIT | 
> > > OPTION_IF_EXISTS | \
> > > +   OPTION_INSERT_HISTORY)
> > >
> > > -/* Shouldn't be defined before */
> > > -#define EXPECTED_OPTIONS \
> > > -  ((1ULL << 14) | (1ULL << 26) | (1ULL << 27) | (1ULL << 19) | (1ULL << 
> > > 28))
> > > -
> > > -#if OPTIONS_WRITTEN_TO_BIN_LOG != EXPECTED_OPTIONS
> > > -#error OPTIONS_WRITTEN_TO_BIN_LOG must NOT change their values!
> > > +#if OPTIONS_WRITTEN_TO_BIN_LOG >= (1ULL << 32)
> > > +#error OPTIONS_WRITTEN_TO_BIN_LOG must NOT exceed 32 bits!
> >
> > Well, no. This check makes sure that OPTION_RELAXED_UNIQUE_CHECKS
> > is 1<<27, that OPTION_AUTO_IS_NULL is 1<<14, etc.
> 
> Reverted.

I'm sorry, but that my comment above is now obsolete.

For explicit_defaults_for_timestamp feature I've removed this whole
EXPECTED_OPTIONS define. Its purpose was to ensure that the set of
"expected options" never changes, it's hard-coded for replication to
work. Obviously, assuming it'll never change was a bit... optimistic.
Monty already changed it by adding OPTION_IF_EXISTS, you changed it for
OPTION_INSERT_HISTORY, and so did I for explicit_defaults_for_timestamp.

So now there is no EXPECTED_OPTIONS define, the set of
OPTIONS_WRITTEN_TO_BIN_LOG can grow. You can add new flags to it, but
you need to record the version when you did it in the method
Format_description_log_event::deduct_options_written_to_bin_log().

Like

  if (server_version_split < Version(10,11,0))
return;
  options_written_to_bin_log|= OPTION_INSERT_HISTORY;

See commit 7b500f04fb0b.

Rebase on top of the latest 10.6 to get 7b500f04fb0b and the
deduct_options_written_to_bin_log() method.

> > > diff --git a/sql/handler.cc b/sql/handler.cc
> > > index 393f6234653..4bbb40abb5b 100644
> > > --- a/sql/handler.cc
> > > +++ b/sql/handler.cc
> > > @@ -7493,6 +7496,26 @@ int handler::ha_write_row(const uchar *buf)
> > >DBUG_RETURN(error);
> > >}
> > >
> > > +  if (table->versioned() && !table->vers_write)
> > > +  {
> > > +Field *row_start= table->vers_start_field();
> > > +Field *row_end= table->vers_end_field();
> > > +MYSQL_TIME ltime;
> > > +
> > > +bitmap_set_bit(table->read_set, row_start->field_index);
> > > +bitmap_set_bit(table->read_set, row_end->field_index);
> > > +
> > > +/*
> > > +   Inserting the history row directly, check ROW_START <= ROW_END and
> > > +   ROW_START is non-zero.
> > > +*/
> > > +if (!row_end->is_max() && (
> > > +  (row_start->cmp(row_start->ptr, row_end->ptr) >= 0) ||
> > > +  row_start->get_date(, Datetime::Options(
> > > +TIME_NO_ZERO_DATE, 
> > > time_round_mode_t(time_round_mode_t::FRAC_NONE)
> >
> > I don't quite understand this condition.
> > checking for row_end->is_max() is redundant, because row_start->cmp()
> > on itself is sufficient. So, I suppose you've done it as an optimization?
> > to avoid expensive cmp()?
> > But in that case you also allow zero date in the row_start, and this looks
> > like a bug.
> > I'd suggest to simplify the code and to remove is_max check.
> > Unless you've run benchmarks and it really helps. And unless
> > I've misunderstood its purpose.
> 
> No, you're right. We can insert not only history but current data and
> row_start there must be checked too. Also, please note this comment:
> 
> # NOTE: having row_start=0 might be useful and can mean
> # "there is no information on when the history was started" (an opposite to 
> row_end=MAX_TIMESTAMP)
> 
> Maybe we should allow it? Just to make the user not invent some values
> for this purpose (like "1970-01-01 00:00:00").

This would violate the concept that  direct inserts are *only* a
usability shortcut and does not allow to do anything new. One cannot
have row_start=0 with

 set timestampt=xxx;
 insert ... values ...();

any other value of row_start can be faked, but not 0.

> > > +  DBUG_RETURN(HA_ERR_WRONG_ROW_TIMESTAMP);
> > > +  }
> > > +
> > >MYSQL_INSERT_ROW_START(table_share->db.str, 
> > > table_share->table_name.str);
> > >mark_trx_read_write();
> > >increment_statistics(::ha_write_count);

Also I've looked at your "Review 2" commits in the bb-10.6-midenok branch,
no new comments.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : 

Re: [Maria-developers] a507f126b03: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability

[Sergei Golubchik]

Hi, Alexander,

> > -ERROR HY000: Illegal parameter data types inet6 and int for operation 'SET'
> > +ERROR HY000: Unknown CAST(int AS inet6) in assignment of 'target'

sorry,  forgot to add, let's also add a db/table name here? and use
the proper backtick quoting for identifiers

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] a507f126b03: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability

[Sergei Golubchik]

Hi, Alexander,

a thought about the error message text and a couple of questions
about changes in results, please, see below:

On Aug 05, Alexander Barkov wrote:
> revision-id: a507f126b03 (mariadb-10.7.4-39-ga507f126b03)
> parent(s): 97d16c7544c
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-08-05 09:42:46 +0400
> message:
> 
> MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, 
> breaks usability
> 
> 1. Store assignment failures on incompatible data types now cause errors if:
> - STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and
> - IGNORE is not used
> 
> Otherwise, only a warning is raised and the statement continues.
> 
> 2. Changing the error/warning test as follows:
> 
> -ERROR HY000: Illegal parameter data types inet6 and int for operation 'SET'
> +ERROR HY000: Unknown CAST(int AS inet6) in assignment of 'target'

Better not to write 'CAST(int AS inet6)' if there's no explicit CAST()
function, I think it might be confusing. I'd say

  Cannot cast int as inet6 in assignment of 'target'

> so in case of a big table it's easier to see which column has the problem.
> The new error text is aslo applied to SP variables.

s/aslo/also/

> diff --git a/mysql-test/main/sp-vars.result b/mysql-test/main/sp-vars.result
> index 127a0993e2b..feef67ccb1d 100644
> --- a/mysql-test/main/sp-vars.result
> +++ b/mysql-test/main/sp-vars.result
> @@ -1026,11 +1026,11 @@ BEGIN
>  SELECT arg;
>  END|
>  CALL p1((1, 2));
> -ERROR HY000: Illegal parameter data types tinyint and row for operation 'SET'
> +ERROR 21000: Operand should contain 1 column(s)

Why not "Unknown CAST" like elsewhere?

> diff --git a/mysql-test/main/type_geometry_mix_int.result 
> b/mysql-test/main/type_geometry_mix_int.result
> index e5d00c63727..0d0a21a8d29 100644
> --- a/mysql-test/main/type_geometry_mix_int.result
> +++ b/mysql-test/main/type_geometry_mix_int.result
> @@ -6,6 +6,7 @@
>  #
>  CREATE TABLE t1 (target GEOMETRY DEFAULT POINT(1,1), source INT DEFAULT 0);
>  # Start of type_store_assignment_incompatible.inc
> +SET @sql_mode_save= (SELECT @@sql_mode);

strange, why not `SET @sql_mode_save= @@sql_mode` ?

>  SET @source_type= (SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS
>  WHERE COLUMN_NAME='source'
> AND TABLE_NAME='t1'
> @@ -47,6 +52,7 @@ INSERT INTO t3 VALUES
>  (2,
>  (SELECT source FROM t2 ORDER BY id LIMIT 1),
>  (SELECT source FROM t2 ORDER BY id LIMIT 1));
> +ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY 
> field

What's that? It wasn't an error, and now it is?

>  SELECT * FROM v3;
>  id   target  source
>  1POINT(1 1)  0

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] f3fc58149dd: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability

[Sergei Golubchik]

Hi, Alexander,

A couple of comments:

On Aug 03, Alexander Barkov wrote:
> revision-id: f3fc58149dd (mariadb-10.7.4-39-gf3fc58149dd)
> parent(s): 97d16c7544c
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-08-03 11:28:31 +0400
> message:
> 
> MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, 
> breaks usability
> 
> Store assignment failures on incompatible data types now cause errors if:
> - STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and
> - IGNORE is not used
> 
> Otherwise, only a warning is raised and the statement continues.
> 
> TODO: add MTR tests for "loose" modes.
> 
> diff --git a/sql/field.cc b/sql/field.cc
> index 249269a6b1d..e997e2cbd69 100644
> --- a/sql/field.cc
> +++ b/sql/field.cc
> @@ -964,7 +964,7 @@ Type_handler::aggregate_for_result_traditional(const 
> Type_handler *a,
>  }
>  
>  
> -bool Field::check_assignability_from(const Type_handler *from) const
> +bool Field::check_assignability_from(const Type_handler *from, bool error) 
> const

"error" is very confusing here, I had to read the whole patch to
understand what it means. Better names:
* only_warning
* ignore
* strict
* issue_error

>  {
>/*
>  Using type_handler_for_item_field() here to get the data type handler
> @@ -982,6 +982,15 @@ bool Field::check_assignability_from(const Type_handler 
> *from) const
>type_handler_for_item_field());
>if (th.aggregate_for_result(from->type_handler_for_item_field()))
>{
> +if (!error)
> +{
> +  THD *thd= get_thd();
> +  push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN,
> +   ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION,
> +   ER_THD(thd, 
> ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION),
> +   type_handler()->name().ptr(), from->name().ptr(), "SET");
> +  return false;
> +}
>  my_error(ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, MYF(0),
>   type_handler()->name().ptr(), from->name().ptr(), "SET");
>  return true;

if you wouldn't want to change the error message in a followup commit,
I'd suggested something like

  my_error(ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION,
   MYF(error ? 0 : ME_WARNING),
   type_handler()->name().ptr(), from->name().ptr(), "SET");
  return error;

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Review: MDEV-29075 Changing explicit_defaults_for_timestamp within stored procedure works inconsistently

[Sergei Golubchik]

Hi, Alexander,

On Jul 29, Alexander Barkov wrote:
>Hello Sergei,
> 
> I have a couple of suggestions:
> 
> > commit 7b8304045272111a6f4d44196d6b37cbfef06f37
> > Author: Sergei Golubchik 
> > Date:   Wed Jul 20 17:31:48 2022 +0200
> > 
> > MDEV-29075 Changing explicit_defaults_for_timestamp within stored 
> > procedure works inconsistently
> > 
> > move the OPTION_EXPLICIT_DEF_TIMESTAMP check from the parsing step
> > to the execution
> 
> > > diff --git a/sql/field.cc b/sql/field.cc
> > index 2e0c70d3d13..23ebb07b7f7 100644
> > --- a/sql/field.cc
> > +++ b/sql/field.cc
> > @@ -10824,6 +10824,7 @@ Column_definition::Column_definition(
> >comment=old_field->comment;
> >vcol_info=  old_field->vcol_info;
> >option_list= old_field->option_list;
> > +  explicitly_nullable= !(old_field->flags & NOT_NULL_FLAG);
> 
> This introduces asymmetry in how NULL and NOT NULL attributes are handled.
> 
> I suggest considering these ways:
> 
> 1. Change "bool explicitly_nullable" to "bool explicit_nullability",
> which will be:
> 
> - false if nothing was specified
> - true if NULL or NOT NULL was specified.

I've done that, see commit 7120195ac1c,
but I don't like how it looks. When NOT_NULL_FLAG is set, the value is
redundant, it's not used anywhere and only adds few pointless
assignments to sql_yacc.yy.

We only need to know how to interpret the case when NOT_NULL_FLAG is not
set, that's what explicitly_nullable was doing.

> 2. Or don't store NOT_NULL_FLAG in Column_definition::flags at all.
> 
> Add a new Column_definition enum member with three values for:
> - not specified
> - explicit NULL
> - explicit NOT NULL
> 
> As discussed on slack, the latter will need some changes.
> But I think not that much, as only Column_definition needs changes,
> while Field does not seem to need at this point.

I don't know how to do it in a robust way. Column_definition and Field
have both the same flags member, one typically just copies it back and
forth. Having just one bit out of Column_definition::flags is very
error-prone, I'd prefer all flags to be stored very differently in
Column_definition, so that one couldn't just copy them. But this
explicitly_nullable thing cannot justify such a big change.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] review for (MDEV-28632) bugfix: DEFAULT NULL was allowed for NOT NULL columns

[Sergei Golubchik]

Hi, Alexander,

I added tests and created an MDEV.
Specific replies below:

On Jul 29, Alexander Barkov wrote:
>Hello Sergei,
> 
> > commit 5a362d486b30fdaf3c7a360737331767154b4ee8
> > Author: Sergei Golubchik 
> > Date:   Mon Jul 18 22:53:27 2022 +0200
> > 
> > bugfix: DEFAULT NULL was allowed for NOT NULL columns
> > 
> > this was detected during parsing (so NOT NULL DEFAULT NULL was
> > an error), but if a column was made NOT NULL later (e.g. as a
> > part of primary key, in mysql_prepare_create_table()), DEFAULT
> > NULL was accepted and ignored.
> > 
> > to fix this the NOT NULL DEFAULT NULL was moved into
> > mysql_prepare_create_table().
> 
> I'm not sure that in case of auto_increment we
> should really disallow DEFAULT NULL under terms of this TIMESTAMP task.
> 
> Look at this script:
> 
> DROP TABLE IF EXISTS t1;
> CREATE TABLE t1 (a INT NOT NULL AUTO_INCREMENT PRIMARY KEY);
> INSERT INTO t1 VALUES (NULL);
> Query OK, 1 row affected (0.012 sec)
> 
> NULL can be inserted without errors.

Well, you don't even need AUTO_INCREMENT for that:

  create table t1 (a int not null);
  create trigger trg1 before insert on t1 for each row set new.a=1;
  insert into t1 values (null);

NULL can be inserted without errors :)

> > @@ -3647,6 +3643,15 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO 
> > *create_info,
> >{
> >  Field::utype type= (Field::utype) MTYP_TYPENR(sql_field->unireg_check);
> >  
> > +if (sql_field->default_value &&
> > +sql_field->default_value->expr->type() == Item::NULL_ITEM &&
> > +sql_field->flags & NOT_NULL_FLAG &&
> > +!(sql_field->flags & AUTO_INCREMENT_FLAG))
> > +{
> > +  my_error(ER_INVALID_DEFAULT, MYF(0), sql_field->field_name.str);
> > +  DBUG_RETURN(TRUE);
> > +}
> 
> ^^^ now obvious errors are caught only during EXECUTE,
> although it's already clear at the PREPARE stage that the DEFAULT is wrong:
> 
> MariaDB [test]> PREPARE stmt FROM 'CREATE TABLE T1 (a INT NOT NULL 
> DEFAULT NULL)';
> Query OK, 0 rows affected (0.000 sec)
> Statement prepared
> 
> MariaDB [test]> EXECUTE stmt;
> ERROR 1067 (42000): Invalid default value for 'a'
> 
> I suggest we still catch non-unambiguous cases
> (when nothing depends on system variables) during PREPARE.
> Inside some existing or a new Type_handler method.

it's quite tricky. At most I can restore the old check and _some_ cases
will be reported at prepare (explicit NOT NULL DEFAULT NULL, that is).

But DEFAULT NULL PRIMARY KEY can only be detected at execute, as
PRIMARY KEY -> NOT NULL adjustment happens only then.

> It seems this code needs to be changed:
> > /*
> >   Set NO_DEFAULT_VALUE_FLAG if this field doesn't have a default value 
> > and
> >   it is NOT NULL, not an AUTO_INCREMENT field, not a TIMESTAMP and not
> >   updated trough a NOW() function.
> > */
> > if (!sql_field->default_value &&
> > !sql_field->has_default_function() &&
> > (sql_field->flags & NOT_NULL_FLAG) &&
> > (!sql_field->is_timestamp_type() ||
> >  (thd->variables.option_bits & OPTION_EXPLICIT_DEF_TIMESTAMP))&&
> > !sql_field->vers_sys_field())
> > {
> >   sql_field->flags|= NO_DEFAULT_VALUE_FLAG;
> >   sql_field->pack_flag|= FIELDFLAG_NO_DEFAULT;
> > }

nope, works fine. I've added tests.

> > diff --git a/storage/spider/mysql-test/spider/bg/t/spider_fixes.test 
> > b/storage/spider/mysql-test/spider/bg/t/spider_fixes.test
> > index b222f494ba1..41f87e61416 100644
> > --- a/storage/spider/mysql-test/spider/bg/t/spider_fixes.test
> > +++ b/storage/spider/mysql-test/spider/bg/t/spider_fixes.test
> > @@ -294,14 +294,14 @@ if ($USE_CHILD_GROUP2)
> >  --disable_query_log
> >  echo CREATE TABLE ta_l (
> >a int(11) NOT NULL DEFAULT '0',
> > -  b char(1) DEFAULT NULL,
> > -  c datetime DEFAULT NULL,
> > +  b char(1),
> > +  c datetime,
> >PRIMARY KEY (a, b, c)
> >  ) MASTER_1_ENGINE MASTER_1_CHARSET MASTER_1_COMMENT5_2_1;
> >  eval CREATE TABLE ta_l (
> >a int(11) NOT NULL DEFAULT '0',
> > -  b char(1) DEFAULT NULL,
> > -  c datetime DEFAULT NULL,
> > +  b char(1),
> > +  c datetime,
> >PRIMARY KEY (a, b, c)
> >  ) $MASTER_1_ENGINE $MASTER_1_CHARSET $MASTER_1_COMMENT5_2_1;
> >  --enable_query_log
> 
> Why this change ^^^ ?

PRIMARY KEY makes columns NOT NULL.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ce5cc8fb905: MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added

[Sergei Golubchik]

Hi, Nikita,

Sure, as you like

On Jul 13, Nikita Malyavin wrote:
> I did, in rpl_alter_extra_persistent.test.
> 
> Btw, maybe also rename this file, into something not referring to
> PERSISTENT?
> 
> On Wed, 13 Jul 2022 at 23:41, Sergei Golubchik  wrote:
> 
> > Hi, Nikita,
> >
> > Please, add a test case for DEFAULT in replication, something like I
> > described below.
> >
> > On Jul 13, Nikita Malyavin wrote:
> > > On Thu, 7 Jul 2022 at 16:05, Sergei Golubchik  wrote:
> > >
> > > > Hi, Nikita,
> > > >
> > > > This is good, but I think fill_extra_persistent_columns() shouldn't be
> > > > used at all.
> > > >
> > > > It doesn't handle default values. I've changed
> > rpl_alter_extra_persistent
> > > > test
> > > > as
> > > >
> > > > -alter table t1 add column z1 int as(a+1) virtual, add column z2 int as
> > > > (a+2) persistent;
> > > > +alter table t1 add column z1 int as(a+1) virtual, add column z2 int
> > > > default (a+2);
> > > >
> > > > and, of course, new column z2 wasn't updated properly.
> > > > An easy fix would be to use your code in all cases and remove
> > > > fill_extra_persistent_columns().
> > > >
> > > > Could you do that too, please? And add the test case too. May be just,
> > > > append
> > > >
> > > >, add column z3 int default (a+2)
> > > >
> > > I'm afraid that we still should mark whose fields for write, or we'll
> > > end up broken.
> > > So i guess i'll just remove the function and will add a raw cycle
> > > through thow fields with marking.
> > > There's no error check, so it's not worth a function anymore
> >
> > Regards,
> > Sergei
> > VP of MariaDB Server Engineering
> > and secur...@mariadb.org
> >
> 
> 
> -- 
> Yours truly,
> Nikita Malyavin
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ce5cc8fb905: MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added

[Sergei Golubchik]

Hi, Nikita,

Please, add a test case for DEFAULT in replication, something like I
described below.

On Jul 13, Nikita Malyavin wrote:
> On Thu, 7 Jul 2022 at 16:05, Sergei Golubchik  wrote:
> 
> > Hi, Nikita,
> >
> > This is good, but I think fill_extra_persistent_columns() shouldn't be
> > used at all.
> >
> > It doesn't handle default values. I've changed rpl_alter_extra_persistent
> > test
> > as
> >
> > -alter table t1 add column z1 int as(a+1) virtual, add column z2 int as
> > (a+2) persistent;
> > +alter table t1 add column z1 int as(a+1) virtual, add column z2 int
> > default (a+2);
> >
> > and, of course, new column z2 wasn't updated properly.
> > An easy fix would be to use your code in all cases and remove
> > fill_extra_persistent_columns().
> >
> > Could you do that too, please? And add the test case too. May be just,
> > append
> >
> >, add column z3 int default (a+2)
> >
> I'm afraid that we still should mark whose fields for write, or we'll
> end up broken.
> So i guess i'll just remove the function and will add a raw cycle
> through thow fields with marking.
> There's no error check, so it's not worth a function anymore

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 3e2d297830b: MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique

[Sergei Golubchik]

Hi, Nikita,

On Jul 08, Nikita Malyavin wrote:
> revision-id: 3e2d297830b (mariadb-10.6.1-511-g3e2d297830b)
> parent(s): c1feb8c3c32
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2022-07-05 15:20:04 +0300
> message:
> 
> MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique
> 
> 1. ER_KEY_NOT_FOUND
> Some virtual columns were not updated because were not included in read_set
> to the moment of calculation.
> 
> In Row_logs_event::do_apply_event some fields are excluded based on m_cols
> value, the number of replicated rows.

the number of replicated *columns*

> We can't rely on this. Basically, we'd be satisfied, if all columns were
> just set, at least for now.
> 
> 2. ER_LOCK_WAIT_TIMEOUT
> This is a long unique specific problem.
> 
> Sometimes, lookup_handler is created for to->file. To properly free it,
> ha_reset should be called. It is usually done by calling
> close_thread_table, but ALTER TABLE makes it differently. Hence, a single
> ha_reset call is added to mysql_alter_table.
> 
> Also, the lifetime of lookup_handler is corrected: it's been allocated on
> thd->mem_root, which in case of online alter
> is event's mem_root, and can vary other ways.
> It is changed to table->mem_root, which is more appropriate:
> ha_reset is called only when table is closed, or in a few additional cases,
> which correspond to a statement end. So memory leaks are unlikely here.
> 
> diff --git a/sql/handler.cc b/sql/handler.cc
> index d71c0fc83a3..fbebfd27203 100644
> --- a/sql/handler.cc
> +++ b/sql/handler.cc
> @@ -3212,7 +3212,7 @@ int handler::create_lookup_handler()
>handler *tmp;
>if (lookup_handler != this)
>  return 0;
> -  if (!(tmp= clone(table->s->normalized_path.str, table->in_use->mem_root)))
> +  if (!(tmp= clone(table->s->normalized_path.str, >mem_root)))

that's wrong, the lifetime of a lookup handler is one statement, it's
destroyed at the end. You cannot keep allocating new handlers on the
table->mem_root for every statement.

>  return 1;
>lookup_handler= tmp;
>return lookup_handler->ha_external_lock(table->in_use, F_RDLCK);
> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> index 89ca2a41937..2c19bf67586 100644
> --- a/sql/log_event_server.cc
> +++ b/sql/log_event_server.cc
> @@ -6007,6 +6007,11 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi)
>   if (m_width == table->s->fields && bitmap_is_set_all(_cols))
>set_flags(COMPLETE_ROWS_F);
>  
> +Rpl_table_data rpl_data{};
> +if (rgi) rgi->get_table_data(table, _data);
> +
> +if (!rpl_data.copy_fields)
> +{
>  /* 
>Set tables write and read sets.
>
> @@ -6027,17 +6027,19 @@ int Rows_log_event::do_apply_event(rpl_group_info 
> *rgi)
>  bitmap_set_all(table->read_set);
>  if (get_general_type_code() == DELETE_ROWS_EVENT ||
>  get_general_type_code() == UPDATE_ROWS_EVENT)
>bitmap_intersect(table->read_set,_cols);
>  
>  bitmap_set_all(table->write_set);
>  
>  /* WRITE ROWS EVENTS store the bitmap in m_cols instead of m_cols_ai */
>  MY_BITMAP *after_image= ((get_general_type_code() == UPDATE_ROWS_EVENT) ?
>   _cols_ai : _cols);
>  bitmap_intersect(table->write_set, after_image);
>  
>  this->slave_exec_mode= slave_exec_mode_options; // fix the mode
> +}

I'm sorry, I don't quite understand. "some fields are excluded based on
m_cols value" - what columns were excluded? What was the value of m_cols
here?

>  table->rpl_write_set= table->write_set;
>  
>  // Do event specific preparations
>  error= do_before_row_operations(rli);
> @@ -6045,8 +6052,6 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi)
>extra columns on the slave. In that case, do not force
>MODE_NO_AUTO_VALUE_ON_ZERO.
>  */
> -Rpl_table_data rpl_data{};
> -if (rgi) rgi->get_table_data(table, _data);
>  sql_mode_t saved_sql_mode= thd->variables.sql_mode;
>  if (!is_auto_inc_in_extra_columns())
>thd->variables.sql_mode= (rpl_data.copy_fields ? saved_sql_mode : 0)
> diff --git a/sql/sql_table.cc b/sql/sql_table.cc
> index 29c4d09cd39..a3efcdd9785 100644
> --- a/sql/sql_table.cc
> +++ b/sql/sql_table.cc
> @@ -11011,6 +11011,9 @@ do_continue:;
> !(table->file->ha_table_flags() & HA_REUSES_FILE_NAMES) &&
> !(new_table->file->ha_table_flags() &
>   HA_REUSES_FILE_NAMES));
> +
> +  // Close lookup_handler.
> +  new_table->file->ha_reset();

please, put it not here, but inside the following
thd->drop_temporary_table() call. It'll also fix Sanja's bug MDEV-26433.
And please coordinate with him to put it on the same place so that git
would detect a conflict and we could merge it to have only one reset
call in that method, not two.

>/*
>  Close the intermediate table that will be the new table, but do
>  not delete it! Even though MERGE tables do not 

Re: [Maria-developers] 73c4fd41afc: Do not ignore sql_mode when replicating

[Sergei Golubchik]

Hi, Nikita,

On Jul 07, Nikita Malyavin wrote:
> 
> diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
> index aba846453e9..89ca2a41937 100644
> --- a/sql/log_event_server.cc
> +++ b/sql/log_event_server.cc
> @@ -6045,9 +6045,12 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi)
>extra columns on the slave. In that case, do not force
>MODE_NO_AUTO_VALUE_ON_ZERO.
>  */
> +Rpl_table_data rpl_data{};
> +if (rgi) rgi->get_table_data(table, _data);
>  sql_mode_t saved_sql_mode= thd->variables.sql_mode;
>  if (!is_auto_inc_in_extra_columns())
> -  thd->variables.sql_mode= MODE_NO_AUTO_VALUE_ON_ZERO;
> +  thd->variables.sql_mode= (rpl_data.copy_fields ? saved_sql_mode : 0)

you start using this condition in many places and it's quite cryptic for
those, who don't know the code. I'd suggest to add method, like

 struct Rpl_table_data
 {
   ...
   bool is_online_alter() { return copy_fields != NULL; }
   ...
 };

btw, it fits very well in your Rpl_table_data simplification.

> +   | MODE_NO_AUTO_VALUE_ON_ZERO;

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] ce5cc8fb905: MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added

[Sergei Golubchik]

Hi, Nikita,

This is good, but I think fill_extra_persistent_columns() shouldn't be
used at all.

It doesn't handle default values. I've changed rpl_alter_extra_persistent test
as

-alter table t1 add column z1 int as(a+1) virtual, add column z2 int as (a+2) 
persistent;
+alter table t1 add column z1 int as(a+1) virtual, add column z2 int default 
(a+2);

and, of course, new column z2 wasn't updated properly.
An easy fix would be to use your code in all cases and remove
fill_extra_persistent_columns().

Could you do that too, please? And add the test case too. May be just,
append

   , add column z3 int default (a+2)

On Jul 07, Nikita Malyavin wrote:
> revision-id: ce5cc8fb905 (mariadb-10.6.1-506-gce5cc8fb905)
> parent(s): 49ad875902e
> author: Nikita Malyavin
> committer: Nikita Malyavin
> timestamp: 2022-07-04 15:55:03 +0300
> message:
> 
> MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added
> 
> We shouldn't rely on `fill_extra_persistent_columns`, as it only updates
> fields which have an index > cols->n_bits (replication bitmap width).
> 
> Normal update_virtual_fields+update_default_fields shoudl be done.
> 
> diff --git a/sql/rpl_record.cc b/sql/rpl_record.cc
> index 17731e64600..42f476fc112 100644
> --- a/sql/rpl_record.cc
> +++ b/sql/rpl_record.cc
> @@ -412,6 +412,26 @@ int unpack_row(rpl_group_info *rgi, TABLE *table, uint 
> const colcnt,
>  {
>copy->do_copy(copy);
>  }
> +if (table->default_field)
> +{
> +  error= table->update_default_fields(table->in_use->lex->ignore);
> +  if (unlikely(error))
> +DBUG_RETURN(error);
> +}
> +if (table->vfield)
> +{
> +  error= table->update_virtual_fields(table->file, 
> VCOL_UPDATE_FOR_WRITE);
> +  if (unlikely(error))
> +DBUG_RETURN(error);
> +}
> +  }
> +  else
> +  {
> +/*
> +  Add Extra slave persistent columns
> +*/
> +if (unlikely(error= fill_extra_persistent_columns(table, cols->n_bits)))
> +  DBUG_RETURN(error);
>}
>  
>/*
> @@ -439,12 +459,6 @@ int unpack_row(rpl_group_info *rgi, TABLE *table, uint 
> const colcnt,
>  }
>}
>  
> -  /*
> -Add Extra slave persistent columns
> -  */
> -  if (unlikely(error= fill_extra_persistent_columns(table, cols->n_bits)))
> -DBUG_RETURN(error);
> -
>/*
>  We should now have read all the null bytes, otherwise something is
>  really wrong.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] How to know when a FUNCTION is AGGREGATE?

[Sergei Golubchik]

Hi, Martijn,

It seems there is no way to distinguish, short of parsing the routine
body looking for 'fetch group next row'.

I've reported it as a bug, https://jira.mariadb.org/browse/MDEV-28849

On Jun 15, Martijn Tonies (Upscene Productions) wrote:
> Hi all,
> 
> Since MariaDB 10.3, you can use an AGGREGATE stored function.
> 
> https://mariadb.com/kb/en/stored-aggregate-functions/
> 
> When examining the meta data, how does one distinguish between
> aggregate and normal stored functions?
> 
> Can’t find anything in the information_schema about this.
> 
> 
> With regards,
> 
> Martijn Tonies
> Upscene Productions
> https://www.upscene.com
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 8c2c0e6079a: MDEV-27265 Improve contraction performance in UCA collations

[Sergei Golubchik]

Hi, Alexander,

Why haven't you used HASH?

On Jun 11, Alexander Barkov wrote:
> revision-id: 8c2c0e6079a (mariadb-10.9.1-13-g8c2c0e6079a)
> parent(s): 27fa5c10eff
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-05-26 13:17:34 +0400
> message:
> 
> MDEV-27265 Improve contraction performance in UCA collations
> 
> Adding a hash table for contractions.
> 
> The old code iterated through all items in MY_CONTRACTIONS,
> and was much slower, especially for those contractions
> in the end of the list.
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 49ecf935415: MDEV-27009 Add UCA-14.0.0 collations

[Sergei Golubchik]

Hi, Alexander,

Few comments/questions below.
Meanwhile I'm reviewing bb-10.9-bar-uca14

On May 26, Alexander Barkov wrote:
> 
> By the way, perhaps some of these statements should display
> short collation names:
> 
>SHOW CREATE TABLE t1;
>SHOW CREATE DATABASE db1;
>SELECT COLLATION_NAME FROM INFORMATION_SCHEMA.COLUMNS;
>SELECT TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES;
>SELECT DEFAULT_COLLATION_NAME FROM INFORMATION_SCHEMA.SCHEMATA;
> 
> Can we discuss this?

Short names, I guess. First two - for readability, the last three - so
that one could join with INFORMATION_SCHEMA.COLLATIONS table.

> >> diff --git a/include/m_ctype.h b/include/m_ctype.h
> >> index 4c6628b72b3..706764ead2a 100644
> >> --- a/include/m_ctype.h
> >> +++ b/include/m_ctype.h
> >> @@ -240,6 +242,46 @@ typedef enum enum_repertoire_t
> >>   } my_repertoire_t;
> >>   
> >> +/* ID compatibility */
> >> +typedef enum enum_collation_id_type
> >> +{
> >> +  MY_COLLATION_ID_TYPE_PRECISE=  0,
> >> +  MY_COLLATION_ID_TYPE_COMPAT_100800=1
> >> +} my_collation_id_type_t;
> >> +
> >> +/* Collation name display modes */
> >> +typedef enum enum_collation_name_mode
> >> +{
> >> +  MY_COLLATION_NAME_MODE_FULL= 0,
> >> +  MY_COLLATION_NAME_MODE_CONTEXT=  1
> >> +} my_collation_name_mode_t;
> >> +
> >> +/* Level flags */
> >> +#define MY_CS_LEVEL_BIT_PRIMARY0x00
> >> +#define MY_CS_LEVEL_BIT_SECONDARY  0x01
> >> +#define MY_CS_LEVEL_BIT_TERTIARY   0x02
> >> +#define MY_CS_LEVEL_BIT_QUATERNARY 0x03
> >> +
> >> +#define MY_CS_COLL_LEVELS_S1   (1< >> +
> >> +#define MY_CS_COLL_LEVELS_AI_CS(1< >> +   (1< >> +
> >> +#define MY_CS_COLL_LEVELS_S2   (1< >> +   (1< >> +
> >> +#define MY_CS_COLL_LEVELS_S3   (1< >> +   (1< >> +   (1< > 
> > AI_CS and S3 don't seem to be used yet
> 
> Right, there are no old _AI_CS and _AS_CS (aka S3) collations.
> 
> New _AI_CS and _AS_CS collations definitions
> are initialized by this function:
> 
> my_uca1400_collation_definition_init(MY_CHARSET_LOADER *loader,
>   struct charset_info_st *dst,
>   uint id)
> 
> Level flags are calculated by this function from "id".
> 
> So there are no hard-coded definitions with
> MY_CS_COLL_LEVELS_AI_CS and MY_CS_COLL_LEVELS_S3 either.
> 
> Should I remove these definitions?

as you like, but if you want to keep them - add a comment. otherwise
anyone can remove them when they'll see those defines are unused
 
> >> +
> >> +#define MY_CS_COLL_LEVELS_S4   (1< >> +   (1< >> +   (1< >> +   (1< >> +
> >>   /* Flags for strxfrm */
> >>   #define MY_STRXFRM_LEVEL1  0x0001 /* for primary weights   */
> >>   #define MY_STRXFRM_LEVEL2  0x0002 /* for secondary weights */
> >> diff --git a/strings/ctype-simple.c b/strings/ctype-simple.c
> >> index b579f0af203..d09dfba86ed 100644
> >> --- a/strings/ctype-simple.c
> >> +++ b/strings/ctype-simple.c
> >> @@ -1940,13 +1941,26 @@ my_bool my_propagate_complex(CHARSET_INFO *cs 
> >> __attribute__((unused)),
...
> > 
> > why do you still use the old concept of "strength"? Why not to use
> > bitmap consistently everywhere?
> 
> The collation definition file Index.xml is based on the LDML syntax.
> 
> It uses tags like this:
> 
> 
> 
> This function is needed to handle these LDML tags.
> 
> Btw, to define user-defined _AI_CS collations we'll need
> to add an LDML extension eventually.

Hmm. Are you saying that LDML cannot describe a UCA 14.0.0 collation?

Or L1+L3 without L2 isn't standard?

> >> +}
> >> +
> >> diff --git a/strings/ctype-uca1400data.h b/strings/ctype-uca1400data.h
> >> --- /dev/null
> >> +++ b/strings/ctype-uca1400data.h
> >> @@ -0,0 +1,44151 @@
> >> +/*
> >> +  Generated from allkeys.txt version '14.0.0'
> >> +*/
> > 
> > if it's generated, do you need to check it in?
> > perhaps it should be generated during the build?
> > you've checked in allkeys1400.txt anyway.
> 
> Right, we can consider it.
> 
> Btw, I've checked it all versions:
> 
> $ ls mysql-test/std_data/unicode/
> 
> allkeys1400.txt
> allkeys400.txt
> allkeys520.txt
> 
> So we can generate sources for all three UCA versions
> from these files.
> 
> But I suggest we do it separately.
> Should I create an MDEV for this?

Why would you want to add to the history a file that is generated?
It'll be there forever.

You already have uca-dump.c in the tree, running it during the build is
a few lines in CMakeLists.txt, basically copy-paste, because we already
do it for gen_lex_hash and comp_err.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: 

Re: [Maria-developers] 2fdcfaaba09: MDEV-27832 disable binary logging for SQL SERVICE.

[Sergei Golubchik]

Hi, Alexey,

Looks good. Minor comments below.
Ok to push after addressing them.

On May 06, Alexey Botchkov wrote:
> revision-id: 2fdcfaaba09 (mariadb-10.7.2-8-g2fdcfaaba09)
> parent(s): 33fd136c61b
> author: Alexey Botchkov
> committer: Alexey Botchkov
> timestamp: 2022-05-05 15:03:38 +0400
> message:
> 
> MDEV-27832 disable binary logging for SQL SERVICE.
> 
> Binary logging is now disabled for the queries run by SQL SERVICE.
> The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query.
> 
> diff --git a/mysql-test/suite/plugins/t/test_sql_service.test 
> b/mysql-test/suite/plugins/t/test_sql_service.test
> index b80d78fe6e5..0fd86fccd1a 100644
> --- a/mysql-test/suite/plugins/t/test_sql_service.test
> +++ b/mysql-test/suite/plugins/t/test_sql_service.test
> @@ -58,3 +73,7 @@ drop table t1;
>  
>  uninstall plugin test_sql_service;
>  
> +--replace_column 2 # 5 #
> +--replace_regex /xid=[0-9]+/xid=XX/ /GTID [0-9]+-[0-9]+-[0-9]+/GTID #-#-#/
> +SHOW BINLOG EVENTS LIMIT 3,100;

use --source include/show_binlog_events.inc
It does all the replaces, provides a unified output everywhere, it'll be
updated if show binlog event output changes.
No need to duplicate that.

> +
> diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc
> index e025147c71e..d3b77fbece3 100644
> --- a/sql/sql_prepare.cc
> +++ b/sql/sql_prepare.cc
> @@ -5617,12 +5617,26 @@ class Protocol_local : public Protocol_text
>THD *new_thd;
>Security_context empty_ctx;
>  
> +  my_bool do_log_bin;
> +
>Protocol_local(THD *thd_arg, THD *new_thd_arg, ulong prealloc) :
>  Protocol_text(thd_arg, prealloc),
>  cur_data(0), first_data(0), data_tail(_data), alloc(0),
> -new_thd(new_thd_arg)
> +new_thd(new_thd_arg), do_log_bin(FALSE)
>{}
>   
> +  void set_binlog_vars(my_bool *sav_log_bin)
> +  {
> +*sav_log_bin= thd->variables.sql_log_bin;
> +thd->variables.sql_log_bin= do_log_bin;
> +thd->set_binlog_bit();
> +  }
> +  void restore_binlog_vars(my_bool _log_bin)

this doesn't need to be a reference, just a normal pass-by-value will
do.

> +  {
> +do_log_bin= thd->variables.sql_log_bin;
> +thd->variables.sql_log_bin= sav_log_bin;
> +thd->set_binlog_bit();
> +  }
>  protected:
>bool net_store_data(const uchar *from, size_t length);
>bool net_store_data_cs(const uchar *from, size_t length,

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 4ba5b444b4b: MDEV-28437: Assertion `!eliminated' failed in Item_subselect::exec

[Sergei Golubchik]

Hi, Sergei,

On May 04, Sergei Petrunia wrote:
> revision-id: 4ba5b444b4b (mariadb-10.2.43-97-g4ba5b444b4b)
> parent(s): 70555454b4c
> author: Sergei Petrunia
> committer: Sergei Petrunia
> timestamp: 2022-05-04 12:24:48 +0300
> message:
> 
> MDEV-28437: Assertion `!eliminated' failed in Item_subselect::exec
> 
> (This is the assert that was added in fix for MDEV-26047)
> 
> Table elimination may remove an ON expression from an outer join.
> However SELECT_LEX::update_used_tables() will still call
> 
>   item->walk(::eval_not_null_tables)
> 
> for eliminated expressions. If the subquery is constant and cheap
> Item_cond_and will attempt to evaluate it, which will trigger an
> assert.
> The fix is not to call update_used_tables() or eval_not_null_tables()
> for ON expressions that were eliminated.
> 
> diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc
> index 22ee8801e3a..8e718f2a942 100644
> --- a/sql/sql_lex.cc
> +++ b/sql/sql_lex.cc
> @@ -4229,7 +4229,7 @@ void SELECT_LEX::update_used_tables()
>}
>  }
>  while ((embedding= embedding->embedding));
> -if (tl->on_expr)
> +if (tl->on_expr && !is_eliminated_table(join->eliminated_tables, tl))
>  {
>tl->on_expr->update_used_tables();
>tl->on_expr->walk(::eval_not_null_tables, 0, NULL);
> @@ -4253,8 +4253,11 @@ void SELECT_LEX::update_used_tables()
>if (embedding->on_expr && 
>embedding->nested_join->join_list.head() == tl)
>{

can embedding be eliminated?

if yes - does it make sense to do anything in tl's below that eliminated
embedding? Perhaps it should be (in the do..while above), like

do
{
  bool maybe_null;
  if ((maybe_null= MY_TEST(embedding->outer_join)))
  {
tl->table->maybe_null= maybe_null;
break;
  }
+ if (is_eliminated_table(join->eliminated_tables, embedding))
+   tl= embedding->embedding;
}
while ((embedding= embedding->embedding));

> +if (!is_eliminated_table(join->eliminated_tables, embedding))
> +{
>embedding->on_expr->update_used_tables();
>embedding->on_expr->walk(::eval_not_null_tables, 0, NULL);
> +}
>}
>tl= embedding;
>embedding= tl->embedding;

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 85cc56875e9: MDEV-19398: Assertion `item1->type() == Item::FIELD_ITEM ...

[Sergei Golubchik]

Hi, Sergei,

The solutions looks fine

Two comments below, about the implementation.

ok to push after addressing them

On May 03, Sergei Petrunia wrote:
> revision-id: 85cc56875e9 (mariadb-10.2.43-75-g85cc56875e9)
> parent(s): 3b6c04f44c4
> author: Sergei Petrunia
> committer: Sergei Petrunia
> timestamp: 2022-05-03 14:06:27 +0300
> message:
> 
> MDEV-19398: Assertion `item1->type() == Item::FIELD_ITEM ...
> 
> Window Functions code tries to minimize the number of times it
> needs to sort the select's resultset by finding "compatible"
> OVER (PARTITION BY ... ORDER BY ...) clauses.
> 
> This employs compare_order_elements(). That function assumed that
> the order expressions are Item_field-derived objects (that refer
> to a temp.table). But this is not always the case: one can
> construct queries order expressions are arbitrary item expressions.
> 
> Add handling for such expressions: sort them according to the window
> specification they appeared in.
> This means we cannot detect that two compatible PARTITION BY clauses
> that use expressions can share the sorting step.
> But at least we won't crash.
> can share the same sort
> 
> diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc
> index 989ca0c8803..457849a7569 100644
> --- a/sql/sql_parse.cc
> +++ b/sql/sql_parse.cc
> @@ -8624,6 +8624,7 @@ bool st_select_lex::add_window_def(THD *thd,
>  fields_in_window_functions+= win_part_list_ptr->elements +
>   win_order_list_ptr->elements;
>}
> +  win_def->win_spec_number= window_specs.elements;
>return (win_def == NULL || window_specs.push_back(win_def));
>  }
>  
> @@ -8651,6 +8652,7 @@ bool st_select_lex::add_window_spec(THD *thd,
>   win_order_list_ptr->elements;
>}
>thd->lex->win_spec= win_spec;
> +  win_spec->win_spec_number= window_specs.elements;
>return (win_spec == NULL || window_specs.push_back(win_spec));
>  }
>  
> diff --git a/sql/sql_window.cc b/sql/sql_window.cc
> index 3ef751bc5b9..fbf7aca0197 100644
> --- a/sql/sql_window.cc
> +++ b/sql/sql_window.cc
> @@ -312,15 +312,44 @@ setup_windows(THD *thd, Ref_ptr_array 
> ref_pointer_array, TABLE_LIST *tables,
>  #define CMP_GT 2// Greater then
>  
>  static
> -int compare_order_elements(ORDER *ord1, ORDER *ord2)
> +int compare_order_elements(ORDER *ord1, int weight1,
> +   ORDER *ord2, int weight2)
>  {
>if (*ord1->item == *ord2->item && ord1->direction == ord2->direction)
>  return CMP_EQ;
>Item *item1= (*ord1->item)->real_item();
>Item *item2= (*ord2->item)->real_item();
> -  DBUG_ASSERT(item1->type() == Item::FIELD_ITEM &&
> -  item2->type() == Item::FIELD_ITEM); 
> -  ptrdiff_t cmp= ((Item_field *) item1)->field - ((Item_field *) 
> item2)->field;
> +
> +  bool item1_field= (item1->type() == Item::FIELD_ITEM);
> +  bool item2_field= (item2->type() == Item::FIELD_ITEM);
> +
> +  ptrdiff_t cmp;
> +  if (item1_field && item2_field)
> +cmp= ((Item_field *) item1)->field - ((Item_field *) item2)->field;

1. why is this order stable? I'd rather sort by item->field->field_index
2. please, add an assert, that item1->field->table == item2->field->table

> +  else if (item1_field && !item2_field)
> +return CMP_LT;
> +  else if (!item1_field && item2_field)
> +return CMP_LT;
> +  else
> +  {
> +/*
> +  Ok, item1_field==NULL and item2_field==NULL.
> +  We're not able to compare Item expressions. Order them according to
> +  their passed "weight" (which comes from Window_spec::win_spec_number):
> +*/
> +if (weight1 != weight2)
> +  cmp= weight1 - weight2;
> +else
> +{
> +  /*
> +The weight is the same. That is, the elements come from the same
> +window specification... This shouldn't happen.
> +  */
> +  DBUG_ASSERT(0);
> +  cmp= item1 - item2;
> +}
> +  }
> +
>if (cmp == 0)
>{

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 1c0731e0544: MDEV-26695: Number of an invalid row is not calculated for table value constructor

[Sergei Golubchik]

Hi, Rucha,

Looks good. please also check that row_number values match what's in a
warning, and then ok to push!

On Apr 26, Rucha Deodhar wrote:
> On Tue, Apr 26, 2022 at 12:41 AM Sergei Golubchik  wrote:
> > what if you combine it with union?
> > create table t (a char(1)) values ('a'),('b') union values ('foo');
> >
> > please, add a test for that.
> >
> >
> It gives 3 as row the number (in warning).  Added a test for it.
> Please find it here: 1c0731e0544db4d2f5c107473776bbd6d4d3e2f0
> 
> > On Apr 25, Rucha Deodhar wrote:
> > > revision-id: 1c0731e0544 (mariadb-10.7.1-33-g1c0731e0544)
> > > parent(s): 5e6fd4e8043
> > > author: Rucha Deodhar
> > > committer: Rucha Deodhar
> > > timestamp: 2022-01-23 13:48:18 +0530
> > > message:
> > >
> > > MDEV-26695: Number of an invalid row is not calculated for table value
> > > constructor
> > >
> > > Analysis: counter does not increment while sending rows for table value
> > > constructor and so row_number assumes the default value (0 in this case).
> > > Fix: Increment the counter to avoid counter using default value.
> > >
> > > diff --git a/sql/sql_tvc.cc b/sql/sql_tvc.cc
> > > index bdaf6829fbd..7db0ab217ca 100644
> > > --- a/sql/sql_tvc.cc
> > > +++ b/sql/sql_tvc.cc
> > > @@ -422,7 +422,9 @@ bool table_value_constr::exec(SELECT_LEX *sl)
> > >DBUG_ENTER("table_value_constr::exec");
> > >List_iterator_fast li(lists_of_values);
> > >List_item *elem;
> > > +  THD *cur_thd= sl->parent_lex->thd;
> > >ha_rows send_records= 0;
> > > +  int rc=0;
> > >
> > >if (select_options & SELECT_DESCRIBE)
> > >  DBUG_RETURN(false);
> > > @@ -438,10 +440,10 @@ bool table_value_constr::exec(SELECT_LEX *sl)
> > >
> > >while ((elem= li++))
> > >{
> > > +cur_thd->get_stmt_da()->inc_current_row_for_warning();
> > >  if (send_records >= sl->master_unit()->lim.get_select_limit())
> > >break;
> > > -int rc=
> > > -  result->send_data_with_check(*elem, sl->master_unit(),
> > send_records);
> > > +rc= result->send_data_with_check(*elem, sl->master_unit(),
> > send_records);
> >
> > you don't need to do that anymore (it was needed in your first patch,
> > but not here)
> >
> > Removed.
> 
> > >  if (!rc)
> > >send_records++;
> > >  else if (rc > 0)
> >
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 1c0731e0544: MDEV-26695: Number of an invalid row is not calculated for table value constructor

[Sergei Golubchik]

Hi, Rucha,

what if you combine it with union?

create table t (a char(1)) values ('a'),('b') union values ('foo');

please, add a test for that.

On Apr 25, Rucha Deodhar wrote:
> revision-id: 1c0731e0544 (mariadb-10.7.1-33-g1c0731e0544)
> parent(s): 5e6fd4e8043
> author: Rucha Deodhar
> committer: Rucha Deodhar
> timestamp: 2022-01-23 13:48:18 +0530
> message:
> 
> MDEV-26695: Number of an invalid row is not calculated for table value
> constructor
> 
> Analysis: counter does not increment while sending rows for table value
> constructor and so row_number assumes the default value (0 in this case).
> Fix: Increment the counter to avoid counter using default value.
> 
> diff --git a/sql/sql_tvc.cc b/sql/sql_tvc.cc
> index bdaf6829fbd..7db0ab217ca 100644
> --- a/sql/sql_tvc.cc
> +++ b/sql/sql_tvc.cc
> @@ -422,7 +422,9 @@ bool table_value_constr::exec(SELECT_LEX *sl)
>DBUG_ENTER("table_value_constr::exec");
>List_iterator_fast li(lists_of_values);
>List_item *elem;
> +  THD *cur_thd= sl->parent_lex->thd;
>ha_rows send_records= 0;
> +  int rc=0;
>
>if (select_options & SELECT_DESCRIBE)
>  DBUG_RETURN(false);
> @@ -438,10 +440,10 @@ bool table_value_constr::exec(SELECT_LEX *sl)
>  
>while ((elem= li++))
>{
> +cur_thd->get_stmt_da()->inc_current_row_for_warning();
>  if (send_records >= sl->master_unit()->lim.get_select_limit())
>break;
> -int rc=
> -  result->send_data_with_check(*elem, sl->master_unit(), send_records);
> +rc= result->send_data_with_check(*elem, sl->master_unit(), send_records);

you don't need to do that anymore (it was needed in your first patch,
but not here)

>  if (!rc)
>send_records++;
>  else if (rc > 0)

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 585cd1f52e7: MDEV-26875: Wrong user in SET DEFAULT ROLE error

[Sergei Golubchik]

Hi, Anel,

On Apr 25, Anel Husakovic wrote:
> revision-id: 585cd1f52e7 (mariadb-10.5.14-10-g585cd1f52e7)
> parent(s): 52b32c60c26
> author: Anel Husakovic
> committer: Anel Husakovic
> timestamp: 2022-02-14 13:59:24 +0100
> message:
> 
> MDEV-26875: Wrong user in SET DEFAULT ROLE error
> 
> - Caused by 7c02e8717de5, where 957cb7b7ba35 introduced the bug.
> 
> Reviewed by:
> 
> diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc
> index c4e66cf5d73..e83bc5635dc 100644
> --- a/sql/sql_acl.cc
> +++ b/sql/sql_acl.cc
> @@ -3277,10 +3277,14 @@ static int check_user_can_set_role(THD *thd, const 
> char *user,
>  
> check_role_is_granted_callback,
>  NULL) == -1))
>{
> -/* Role is not granted but current user can see the role */
> -my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted 
> role %`s",
> -MYF(0), thd->security_ctx->priv_user,
> -thd->security_ctx->priv_host, rolename);
> +/* If the SET ROLE is applied on the anonymous user, host is null */
> +if (!host)
> +  my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been 
> granted role %`s",
> +  MYF(0), thd->security_ctx->priv_user, 
> thd->security_ctx->priv_host, rolename);

I don't understand it.
1) what does it mean? why for anonymous user you print priv_user@priv_host ?
2) do you have any tests for that? There are none in the commit

> +else
> +  /* Role is not granted but current user can see the role */
> +  my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been 
> granted role %`s",
> +  MYF(0), user, host, rolename);
>}
>else
>{

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 5f8ffc7271a: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon HANDLER READ

[Sergei Golubchik]

Hi, Rucha,

The fix is ok to push. But as a general comment, better try to define
variables in the smallest scope where they're used. In particular here
it could've been

 {
   MY_BITMAP *old_map= dbug_tmp_use_all_columns(table, 
>write_set);
-  (void) item->save_in_field(key_part->field, 1);
+  int res= item->save_in_field(key_part->field, 1);

because you don't need `res` outside of that scope. Makes the code
easier to read.

Of course, there can be exceptions. For example, if the variable is an
object and its initialization is very expensive.

But whether you move `int res` declaration inside or not - it's ok to
push, no need to ask for another review.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

On Apr 25, Rucha Deodhar wrote:
> revision-id: 5f8ffc7271a (mariadb-10.2.40-251-g5f8ffc7271a)
> parent(s): 2cbf92522b5
> author: Rucha Deodhar
> committer: Rucha Deodhar
> timestamp: 2022-01-27 01:18:19 +0530
> message:
> 
> MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
> upon HANDLER READ
> 
> Analysis: The error state is not stored while checking condition and key name.
> Fix: Return true while checking condition and key name if error is reported
> because geometry object can't be created from the data in the index value for
> HANDLER READ.
> 
> diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc
> index ec3756eceba..7c002a68d9e 100644
> --- a/sql/sql_handler.cc
> +++ b/sql/sql_handler.cc
> @@ -554,6 +554,7 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler,
>  {
>THD *thd= handler->thd;
>TABLE *table= handler->table;
> +  int res= 0;
>if (cond)
>{
>  /* This can only be true for temp tables */
> @@ -613,8 +614,10 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler,
>  if (!in_prepare)
>  {
>MY_BITMAP *old_map= dbug_tmp_use_all_columns(table, 
> >write_set);
> -  (void) item->save_in_field(key_part->field, 1);
> +  res= item->save_in_field(key_part->field, 1);
>dbug_tmp_restore_column_map(>write_set, old_map);
> +  if (res)
> +return 1;
>  }
>  key_len+= key_part->store_length;
>  keypart_map= (keypart_map << 1) | 1;

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Sergei Golubchik]

Hi, Aleksey,

ok to push, thanks!

On Apr 20, Aleksey Midenkov wrote:
> Sergei,
> 
> I updated the branch accordingly. There is your patch fa444975d0f
> where I added log_current_statement().
> 
> On Wed, Apr 20, 2022 at 1:12 PM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey,
> >
> > Okay, sure.
> > I was going to push it after the release.
> > And in 10.3 to simplify future merges.
> >
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Sergei Golubchik]

Hi, Aleksey,

Okay, sure.
I was going to push it after the release.
And in 10.3 to simplify future merges.

On Apr 20, Aleksey Midenkov wrote:
> Hi Sergei!
> 
> On Tue, Apr 19, 2022 at 11:13 AM Sergei Golubchik  wrote:
> >
> > Hi, Aleksey,
> >
> > On Apr 18, Aleksey Midenkov wrote:
> > > >
> > > > I have a second patch that renames flags to have a very well-defined
> > > > meaning and name.
> > > >
> > > >   OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX
> > > >   log_current_statement -> OPTION_BINLOG_THIS_STMT
> > >
> > > Where is your patch?
> >
> > It's really just renaming, almost nothing else.
> > But here it is, attached.
> 
> I'd keep log_current_statement() like this:
> 
> --- a/sql/sql_class.h
> +++ b/sql/sql_class.h
> @@ -3562,4 +3562,8 @@ class THD: public THD_count, /* this must be first */
>   /* set during loop of derived table processing */
>   bool   derived_tables_processing;
>   bool   tablespace_op;/* This is TRUE in DISCARD/IMPORT TABLESPACE */
> +  bool   log_current_statement() const
> +  {
> +return variables.option_bits & OPTION_BINLOG_THIS_STMT;
> +  }
>   /**
> 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Sergei Golubchik]

Hi, Aleksey,

On Apr 18, Aleksey Midenkov wrote:
> >
> > I have a second patch that renames flags to have a very well-defined
> > meaning and name.
> >
> >   OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX
> >   log_current_statement -> OPTION_BINLOG_THIS_STMT
> 
> Where is your patch?

It's really just renaming, almost nothing else.
But here it is, attached.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org
commit 0593fd1926f
Author: Sergei Golubchik 
Date:   Mon Apr 11 22:06:21 2022 +0200

cleanup: log_current_statement and OPTION_KEEP_LOG

rename OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX.
Meaning: transaction cache will be written to binlog even on rollback.

convert log_current_statement to OPTION_BINLOG_THIS_STMT.
Meaning: the statement will be written to binlog (or trx binlog cache)
even if it normally wouldn't be.

setting OPTION_BINLOG_THIS_STMT must always set OPTION_BINLOG_THIS_TRX,
otherwise the statement won't be logged if the transaction is rolled back.
Use OPTION_BINLOG_THIS to set both.

diff --git a/sql/log.cc b/sql/log.cc
index 70ceecc66f8..02776f3b43e 100644
--- a/sql/log.cc
+++ b/sql/log.cc
@@ -2036,7 +2036,7 @@ inline bool is_prepared_xa(THD *thd)
 /*
   We flush the cache wrapped in a beging/rollback if:
 . aborting a single or multi-statement transaction and;
-. the OPTION_KEEP_LOG is active or;
+. the OPTION_BINLOG_THIS_TRX is active or;
 . the format is STMT and a non-trans table was updated or;
 . the format is MIXED and a temporary non-trans table was
   updated or;
@@ -2047,7 +2047,7 @@ static bool trans_cannot_safely_rollback(THD *thd, bool 
all)
 {
   DBUG_ASSERT(ending_trans(thd, all));
 
-  return ((thd->variables.option_bits & OPTION_KEEP_LOG) ||
+  return ((thd->variables.option_bits & OPTION_BINLOG_THIS_TRX) ||
   (trans_has_updated_non_trans_table(thd) &&
thd->wsrep_binlog_format() == BINLOG_FORMAT_STMT) ||
   (thd->transaction->all.has_modified_non_trans_temp_table() &&
@@ -2446,10 +2446,10 @@ static int binlog_savepoint_rollback(handlerton *hton, 
THD *thd, void *sv)
*/
   if (unlikely(thd->wsrep_trx().is_streaming() ||
(trans_has_updated_non_trans_table(thd)) ||
-   (thd->variables.option_bits & OPTION_KEEP_LOG)))
+   (thd->variables.option_bits & OPTION_BINLOG_THIS_TRX)))
 #else
   if (unlikely(trans_has_updated_non_trans_table(thd) ||
-   (thd->variables.option_bits & OPTION_KEEP_LOG)))
+   (thd->variables.option_bits & OPTION_BINLOG_THIS_TRX)))
 #endif /* WITH_WSREP */
   {
 char buf[1024];
diff --git a/sql/slave.cc b/sql/slave.cc
index 3c5b830fbe2..c0080e4df74 100644
--- a/sql/slave.cc
+++ b/sql/slave.cc
@@ -1368,13 +1368,13 @@ static bool sql_slave_killed(rpl_group_info *rgi)
   if (rli->sql_driver_thd->killed || rli->abort_slave)
   {
 /*
-  The transaction should always be binlogged if OPTION_KEEP_LOG is
+  The transaction should always be binlogged if OPTION_BINLOG_THIS_TRX is
   set (it implies that something can not be rolled back). And such
   case should be regarded similarly as modifing a
   non-transactional table because retrying of the transaction will
   lead to an error or inconsistency as well.
 
-  Example: OPTION_KEEP_LOG is set if a temporary table is created
+  Example: OPTION_BINLOG_THIS_TRX is set if a temporary table is created
   or dropped.
 
   Note that transaction.all.modified_non_trans_table may be 1
@@ -1384,7 +1384,7 @@ static bool sql_slave_killed(rpl_group_info *rgi)
 */
 
 if ((thd->transaction->all.modified_non_trans_table ||
- (thd->variables.option_bits & OPTION_KEEP_LOG)) &&
+ (thd->variables.option_bits & OPTION_BINLOG_THIS_TRX)) &&
 rli->is_in_group())
 {
   char msg_stopped[]=
@@ -1396,10 +1396,10 @@ static bool sql_slave_killed(rpl_group_info *rgi)
 "documentation for details).";
 
   DBUG_PRINT("info", ("modified_non_trans_table: %d  OPTION_BEGIN: %d  "
-  "OPTION_KEEP_LOG: %d  is_in_group: %d",
+  "OPTION_BINLOG_THIS_TRX: %d  is_in_group: %d",
   thd->transaction->all.modified_non_trans_table,
   MY_TEST(thd->variables.option_bits & OPTION_BEGIN),
-  MY_TEST(thd->variables.option_bits & 
OPTION_KEEP_LOG),
+  MY_TEST(thd->variables.option_bits & 
OPTION_BINLOG_THIS_TRX),
   rli->is_in_group()));
 
   if (rli->abort_slave)
diff --git a/sql/sql_class.h b/sql/sql_class.h
index 686e6e70766..d68588b4969 100644
--- a/sql/sql_class.

Re: [Maria-developers] 96b40a5e2f5: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_in

[Sergei Golubchik]

Hi, Alexander,

Some comments/questions here. I'll need to look at the patch again after
I understand why you did it this way better.

On Apr 18, Alexander Barkov wrote:
> revision-id: 96b40a5e2f5 (mariadb-10.3.33-61-g96b40a5e2f5)
> parent(s): 7355f7b1f5c
> author: Alexander Barkov
> committer: Alexander Barkov
> timestamp: 2022-04-07 16:22:17 +0400
> message:
>
> MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in 
> row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in 
> row_upd_sec_index_entry (debug) | Corruption

A detailed comment here is a must.
What was the problem, how it was fixed.

> diff --git a/sql/item_func.h b/sql/item_func.h
> index 754b1cd1eb2..d844e45280a 100644
> --- a/sql/item_func.h
> +++ b/sql/item_func.h
> @@ -35,6 +35,36 @@ extern "C" /* Bug in BSDI include 
> file */
>  #include 
>  
>  
> +template
> +FUNC* item_func_create_2_3_args(THD *thd, const LEX_CSTRING ,
> +List *item_list)
> +{
> +  int arg_count= item_list ? item_list->elements : 0;
> +
> +  switch (arg_count) {
> +  case 2:
> +  {
> +Item *param_1= item_list->pop();
> +Item *param_2= item_list->pop();
> +return new (get_thd_memroot(thd)) FUNC(thd, param_1, param_2);
> +  }
> +  case 3:
> +  {
> +Item *param_1= item_list->pop();
> +Item *param_2= item_list->pop();
> +Item *param_3= item_list->pop();
> +return new (get_thd_memroot(thd)) FUNC(thd, param_1, param_2, param_3);

A constructor that takes a List wouldn't need a switch.
And wouldn't need a template either, your create_check_args_ge()
isn't a template.

> +break;
> +  }
> +  default:
> +my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str);
> +break;
> +  }
> +
> +  return NULL;
> +}
> +
> +
>  class Item_func :public Item_func_or_sum
>  {
>void sync_with_sum_func_and_with_field(List );
> @@ -56,8 +86,41 @@ class Item_func :public Item_func_or_sum
>bool check_argument_types_can_return_text(uint start, uint end) const;
>bool check_argument_types_can_return_date(uint start, uint end) const;
>bool check_argument_types_can_return_time(uint start, uint end) const;
> +
> +  void print_schema_name_if_needed(String *to) const
> +  {
> +const LEX_CSTRING schema_name= schema_name_cstring();
> +if (schema_name.length)
> +{
> +  to->append(schema_name);
> +  to->append('.');
> +}

this should check that schema_name isn't the same as the
schema in the path (implicit path, assumed by the current sql_mode).

> +  }
> +  void print_maybe_qualified_name(String *to) const
> +  {
> +print_schema_name_if_needed(to);
> +to->append(func_name());
> +  }
> +
>  public:
>  
> +  // Print an error message for a builtin-schema qualified function call
> +  static void wrong_param_count_error(const LEX_CSTRING _name,
> +  const LEX_CSTRING _name);
> +
> +  // Check that the number of arguments is greater or equal to "expected"
> +  static bool create_check_args_ge(const LEX_CSTRING ,
> +   const List *item_list,
> +   uint expected)
> +  {
> +if (!item_list || item_list->elements < expected)
> +{
> +  my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str);

what's the point of your wrong_param_count_error() if you aren't using it?

> +  return true;
> +}
> +return false;
> +  }
> +
>table_map not_null_tables_cache;
>  
>enum Functype { UNKNOWN_FUNC,EQ_FUNC,EQUAL_FUNC,NE_FUNC,LT_FUNC,LE_FUNC,
> diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc
> index 75abd9906cd..e45fe6ef6d7 100644
> --- a/sql/item_strfunc.cc
> +++ b/sql/item_strfunc.cc
> @@ -3204,6 +3239,39 @@ static String *default_pad_str(String *pad_str, 
> CHARSET_INFO *collation)
>return pad_str;
>  }
>  
> +
> +Item_func_lpad*
> +Item_func_lpad::create(THD *thd, const LEX_CSTRING ,
> +   List *item_list)
> +{
> +  return item_func_create_2_3_args(thd, name, item_list);

may be item_func_create_2_3_args should be in Item_func?
Like create_check_args_ge() is?
and may be create_check_args_ge() should be a template too?

> +}
> +
> +
> +Item_func_lpad_oracle*
> +Item_func_lpad_oracle::create(THD *thd, const LEX_CSTRING ,
> +  List *item_list)
> +{
> +  return item_func_create_2_3_args(thd, name, 
> item_list);
> +}
> +
> +
> +Item_func_rpad*
> +Item_func_rpad::create(THD *thd, const LEX_CSTRING ,
> +   List *item_list)
> +{
> +  return item_func_create_2_3_args(thd, name, item_list);
> +}
> +
> +
> +Item_func_rpad_oracle*
> +Item_func_rpad_oracle::create(THD *thd, const LEX_CSTRING ,
> +  List *item_list)
> +{
> +  return item_func_create_2_3_args(thd, name, 
> item_list);
> +}
> +
> +
>  bool Item_func_pad::fix_length_and_dec()
>  {
>if (arg_count == 3)
> diff --git a/sql/sql_lex.cc 

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Sergei Golubchik]

Hi, Aleksey,

> I attached the patch and this still fails rpl.create_or_replace_row.

small mistake in the patch:

> --- a/sql/sql_class.h
> +++ b/sql/sql_class.h
> @@ -2916,7 +2916,7 @@ class THD: public THD_count, /* this must be first */
>  
>bool binlog_need_stmt_format(bool is_transactional) const
>{
> -return vers_created_partitions && 
> !binlog_get_pending_rows_event(is_transactional);
> +return !binlog_get_pending_rows_event(is_transactional);

if you put here

  log_current_statement && !binlog_get_pending_rows_event(is_transactional)

then all tests pass.

>}
>  
>bool binlog_for_noop_dml(bool transactional_table);

> Why don't you like vers_created_partitions? It might be useful info
> for debugging and for new features.

I have a second patch that renames flags to have a very well-defined
meaning and name.

  OPTION_KEEP_LOG -> OPTION_BINLOG_THIS_TRX
  log_current_statement -> OPTION_BINLOG_THIS_STMT

the first flag means that when a transaction is rolled back, it's
written to binlog with ROLLBACK at the end, not discarded.

The second means that the statement is written to binlog (or binlog trx
cache) even if it otherwise wouldn't be.

So, two flags. Clear names and semantics. "vers_created_partitions" fits
into the second use case, so one should use OPTION_BINLOG_THIS_STMT for it.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Sergei Golubchik]

Hi, Aleksey,

> > Take a look at the bb-10.3-serg-MDEV-24176 branch. Three commits
> > there: two cherry-picks from 10.2 (they shouldn't be pushed into
> > 10.3, instead your commit will eventually be rebased on top of them
> > after they're merged into 10.3) and the "work-in-progress" commit
> > that removes init_lex_with_single_table and
> > CONTEXT_ANALYSIS_ONLY_VCOL_EXPR.
> 
> I just tried to note this is a different bug (as it reproduces in
> master branch). And since it is lower priority maybe we push the
> blocker first?

Okay.
But I have a fix for this different bug already, in bb-10.3-serg-MDEV-24176

So, go ahead and push, then I'll merge with 10.2 and push my fix too.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Sergei Golubchik]

Hi, Aleksey,

On Apr 13, Aleksey Midenkov wrote:
> >
> > No, quite the opposite.
> > I think (see above) that CONTEXT_ANALYSIS_ONLY_VCOL_EXPR
> > (it's set inside init_lex_with_single_table()) is wrong, what you're
> > doing is not "context analysys only", you're preparing items for
> > evaluation.
> 
> That works identically in branch and vanilla 10.3:
> 
> --source include/have_ucs2.inc
> create table t1 (c1 char(1) character set ucs2 collate ucs2_test_ci,
>  v1 char(1) character set ucs2 collate ucs2_test_ci as (c1),
>  v2 int as (c1 = 'b'),
>  v3 int as (v1 = 'b'));
> insert into t1 (c1) values ('a');
> select * from t1 where v1 = 'b';
> drop table t1;
> 
> Result:
> select * from t1 where v1 = 'b';
> c1  v1  v2  v3
> a   a   0   0
> 
> .opt file:
> --character-sets-dir=$MYSQL_TEST_DIR/std_data/ldml/

Wow, this is a very good test. I wonder how you found it.

Yes, it shows, exactly, that it's incorrect to set
CONTEXT_ANALYSIS_ONLY_VCOL_EXPR and then evaluate those items.
Items fixed under CONTEXT_ANALYSIS_ONLY_VCOL_EXPR are *not fit* to be
evaluated.

Take a look at the bb-10.3-serg-MDEV-24176 branch.
Three commits there: two cherry-picks from 10.2
(they shouldn't be pushed into 10.3, instead your commit will eventually
be rebased on top of them after they're merged into 10.3)
and the "work-in-progress" commit that removes
init_lex_with_single_table and CONTEXT_ANALYSIS_ONLY_VCOL_EXPR.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Sergei Golubchik]

Hi, Aleksey,

On Apr 12, Aleksey Midenkov wrote:
> > > > > +bool Vcol_expr_context::init()
> > > > > +{
> > > > > +  /*
> > > > > +  As this is vcol expression we must narrow down name resolution 
> > > > > to
> > > > > +  single table.
> > > > > +  */
> > > > > +  if (init_lex_with_single_table(thd, table, ))
> > > >
> > > CURRENT_TEST: gcol.gcol_bugfixes
> > > mysqltest: At line 579: query 'INSERT INTO t1 (suppliersenttoday)
> > > VALUES (0)' failed: 2013: Lost connection to MySQL server during query
> >
> > this one crashes. on the next line, 580, though.
> > because you set CONTEXT_ANALYSIS_ONLY_VCOL_EXPR, so
> > Type_std_attributes::agg_item_set_converter does not wrap items
> > in Item_func_conv_charset. Which is likely incorrect, because items
> > without wrapping cannot be properly evaluated, and it looks like they
> > has to be evaluated later, so it's not "context analysys only".
> >
> > Crash on wrapped items is https://jira.mariadb.org/browse/MDEV-25638
> > that Sanja is looking at right now.
> 
> Agree, I have the same picture now. Previous faults were on work in
> progress. So it faults anyway and we keep
> init_lex_with_single_table(), right?

No, quite the opposite.
I think (see above) that CONTEXT_ANALYSIS_ONLY_VCOL_EXPR
(it's set inside init_lex_with_single_table()) is wrong, what you're
doing is not "context analysys only", you're preparing items for
evaluation.

Let's wait for Sanja to close his MDEV-25638, and then I'll check this
test case again.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated

[Sergei Golubchik]

Hi, Aleksey,

Basically, it's fine, but I still didn't like that there's a new flag in
THD to mark a statement that has to be logged. So I tried to change to
log_current_statement. I know you tried that and you told me what tests
fails, rpl.create_or_replace_*. I debugged that a bit, and I think the
problem is that select_create inherits from select_insert, so
CREATE ... SELECT runs your code that is only supposed to be run for
INSERT ODKU or REPLACE. This change makes all tests pass (after removing
vers_created_partitions):

@@ -4225,7 +4225,8 @@ bool select_insert::prepare_eof()
   thd->clear_error();
 else
   errcode= query_error_code(thd, killed_status == NOT_KILLED);
-StatementBinlog stmt_binlog(thd, 
thd->binlog_need_stmt_format(trans_table));
+StatementBinlog stmt_binlog(thd, !can_rollback_data() &&
+thd->binlog_need_stmt_format(trans_table));
 res= thd->binlog_query(THD::ROW_QUERY_TYPE,
thd->query(), thd->query_length(),
trans_table, FALSE, FALSE, errcode);

On Apr 10, Aleksey Midenkov wrote:
> revision-id: 17548c8a8b6 (mariadb-10.6.1-250-g17548c8a8b6)
> parent(s): 6282e025a0e
> author: Aleksey Midenkov
> committer: Aleksey Midenkov
> timestamp: 2022-03-07 23:49:35 +0300
> message:
> 
> MDEV-25477 Auto-create breaks replication when triggering event was not 
> replicated
> 
> diff --git a/sql/sql_class.h b/sql/sql_class.h
> index 686e6e70766..4fed1bf7355 100644
> --- a/sql/sql_class.h
> +++ b/sql/sql_class.h
> @@ -2914,6 +2914,13 @@ class THD: public THD_count, /* this must be first */
>int binlog_flush_pending_rows_event(bool stmt_end, bool is_transactional);
>int binlog_remove_pending_rows_event(bool clear_maps, bool 
> is_transactional);
>  
> +  bool binlog_need_stmt_format(bool is_transactional) const
> +  {
> +return vers_created_partitions && 
> !binlog_get_pending_rows_event(is_transactional);
> +  }
> +
> +  bool binlog_for_noop_dml(bool transactional_table);
> +
>/**
>  Determine the binlog format of the current statement.
>  
> @@ -3557,6 +3564,7 @@ class THD: public THD_count, /* this must be first */
>bool   tablespace_op;  /* This is TRUE in DISCARD/IMPORT TABLESPACE */
>/* True if we have to log the current statement */
>boollog_current_statement;
> +  uint   vers_created_partitions;
>/**
>  True if a slave error. Causes the slave to stop. Not the same
>  as the statement execution error (is_error()), since
> diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc
> index 707b8a0d3bf..25551dfa197 100644
> --- a/sql/sql_insert.cc
> +++ b/sql/sql_insert.cc
> @@ -4223,6 +4225,7 @@ bool select_insert::prepare_eof()
>thd->clear_error();
>  else
>errcode= query_error_code(thd, killed_status == NOT_KILLED);
> +StatementBinlog stmt_binlog(thd, 
> thd->binlog_need_stmt_format(trans_table));
>  res= thd->binlog_query(THD::ROW_QUERY_TYPE,
> thd->query(), thd->query_length(),
> trans_table, FALSE, FALSE, errcode);

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] e3e59c9e38d: MDEV-28266 Crash in Field_string::type_handler when calling procedures

[Sergei Golubchik]

Hi, Oleksandr,

On Apr 10, Oleksandr Byelkin wrote:
> revision-id: e3e59c9e38d (mariadb-10.6.6-83-ge3e59c9e38d)
> parent(s): 4e1ca388381
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2022-04-08 10:37:17 +0200
> message:
> 
> MDEV-28266 Crash in Field_string::type_handler when calling procedures
> 
> In case of only metadata request the cursor lack the call to redirect
> Field::orig_table to the temporary table.
> 
> diff --git a/sql/sql_cursor.cc b/sql/sql_cursor.cc
> index 0add5845558..155346341f1 100644
> --- a/sql/sql_cursor.cc
> +++ b/sql/sql_cursor.cc
> @@ -183,6 +183,16 @@ int mysql_open_cursor(THD *thd, select_result *result,
>{
>  Materialized_cursor *materialized_cursor=
>result_materialize->materialized_cursor;
> +if (result_materialize->view_structure_only())
> +{
> +  /*
> +If it was view_structure_only then only send metadata call was made
> +and there was no other call. So we have to "close" the cursor
> +explicitly (which redirect orig_table in the fields, and untie
> +it with closed tables)
> +  */
> +  materialized_cursor->on_table_fill_finished();
> +}
>
> if ((rc= materialized_cursor->open(0)))
> {
>   delete materialized_cursor;
>   goto end;
> }

normally on_table_fill_finished() is called from
Select_materialize::send_eof() and
Select_materialize::abort_result_set().

Note that directly after your added code there's
materialized_cursor->open(), and there we have

  int Materialized_cursor::open(JOIN *join __attribute__((unused)))
  {
...
if (!rc)
{
  thd->server_status|= SERVER_STATUS_CURSOR_EXISTS;
  result->send_eof();
}
else
{
  result->abort_result_set();
}

That is, one could expect that even without your added code,
on_table_fill_finished() would be called.

The problem is that result here is sp_cursor::Select_fetch_into_spvars,
and sp_cursor::Select_fetch_into_spvars::send_eof() is simply
{ return FALSE; }

So, as a fix I'd suggest to remove on_table_fill_finished() from
send_eof/abort_result_set, and put it unconditionally after the if(),
to make it work no matter what select_result was chosen.

Like this:
==
--- a/sql/sql_cursor.cc
+++ b/sql/sql_cursor.cc
@@ -80,19 +80,7 @@ class Select_materialize: public select_unit
   Select_materialize(THD *thd_arg, select_result *result_arg):
 select_unit(thd_arg), result(result_arg), materialized_cursor(0) {}
   virtual bool send_result_set_metadata(List , uint flags);
-  bool send_eof()
-  {
-if (materialized_cursor)
-  materialized_cursor->on_table_fill_finished();
-return false;
-  }
-
-  void abort_result_set()
-  {
-if (materialized_cursor)
-  materialized_cursor->on_table_fill_finished();
-  }
-
+  bool send_eof() { return false; }
   bool view_structure_only() const
   {
 return result->view_structure_only();
@@ -333,6 +321,8 @@ int Materialized_cursor::open(JOIN *join
 result->abort_result_set();
   }

+  on_table_fill_finished();
+
   return rc;
 }
==

with the commit comment saying that "on_table_fill_finished() should
always be done at the end of open(), even if result is not
Select_materialize, but, for example, Select_fetch_into_spvars"

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 62d3496969d: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Sergei Golubchik]

Hi, Aleksey,

This is a review of `git diff 5fbbdbc85ba3 62d3496969db`

Basically, no new comments, see below:

On Apr 07, Aleksey Midenkov wrote:

> diff --git a/libmariadb b/libmariadb
> index f6c3d9fd2af..735a7299dba 16
> --- a/libmariadb
> +++ b/libmariadb
> @@ -1 +1 @@
> -Subproject commit f6c3d9fd2af5d17db64cc996574aa312efd70fcf
> +Subproject commit 735a7299dbae19cc2b82b9697becaf90e9b43047

you've mistakenly checked in a submodule

> diff --git a/sql/field.cc b/sql/field.cc
> index f18fb25ebe3..04c492e6c69 100644
> --- a/sql/field.cc
> +++ b/sql/field.cc
> @@ -2416,6 +2416,13 @@ int Field::set_default()
>if (default_value)
>{
>  Query_arena backup_arena;
> +/*
> +  TODO: this imposes memory leak until table flush when save_in_field()
> +does expr_arena allocation. F.ex. case from main.default:
> +
> +CREATE TABLE t1 (a INT DEFAULT CONCAT('1 '));
> +INSERT INTO t1 VALUES (DEFAULT);

I wasn't able to repeat that. In your branch (didn't try in vanilla 10.3) and
for the above test case I see no additional allocations in table->mem_root
after the table is opened.

> +*/
>  table->in_use->set_n_backup_active_arena(table->expr_arena, 
> _arena);
>  int rc= default_value->expr->save_in_field(this, 0);
>  table->in_use->restore_active_arena(table->expr_arena, _arena);
> diff --git a/sql/table.cc b/sql/table.cc
> index 08d91678b25..d22da17c31a 100644
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -2970,36 +2988,133 @@ static bool fix_vcol_expr(THD *thd, 
> Virtual_column_info *vcol)
...
> +bool Vcol_expr_context::init()
> +{
> +  /*
> +  As this is vcol expression we must narrow down name resolution to
> +  single table.
> +  */
> +  if (init_lex_with_single_table(thd, table, ))

I've already asked about it in a separate email,
so here I'm just marking a line that we haven't fully agreed on yet

> +  {
> +my_error(ER_OUT_OF_RESOURCES, MYF(0));
> +table->map= old_map;
> +return true;
> +  }
> +
> +  lex.sql_command= old_lex->sql_command;
> +  thd->variables.sql_mode= 0;
> +
> +  TABLE_LIST const *tl= table->pos_in_table_list;
> +  DBUG_ASSERT(table->pos_in_table_list);
> +
> +  if (table->pos_in_table_list->security_ctx)
>  thd->security_ctx= tl->security_ctx;
> -  bool res= fix_session_vcol_expr(thd, vcol);
> +
> +  inited= true;
> +  return false;
> +}
> +
 
Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()

[Sergei Golubchik]

Hi, Aleksey,

Thanks. I'll send a separate review email, there will be only replies
here:

On Apr 05, Aleksey Midenkov wrote:
> Hi, Sergei!
> 
> > > @@ -5709,8 +5708,6 @@ bool Item_field::fix_fields(THD *thd, Item 
> > > **reference)
> > >}
> > >  #endif
> > >fixed= 1;
> > > -  if (field->vcol_info)
> > > -fix_session_vcol_expr_for_read(thd, field, field->vcol_info);
> >
> > Did you revert the optimization from 7450cb7f69db?
> > It was "re-fix vcols on demand, not always for every SELECT"
> >
> > and as far as I can see now you again always re-fix everything,
> > is that so?
> 
> It was your suggestion in the previous review to refix always
> (therefore everything). The original fix worked with
> fix_session_vcol_expr_for_read(). But the current fix is simpler and
> cleaner. Was there a real performance problem or was it a theoretical
> issue?

I checked previous emails and wasn't able to find where I suggested to
refix always.

Anyway, it was indeed a theoretical issue and I have no proofs to claim
that that "optimization" made a difference. So, okay, feel free to refix
always.

> > >if (thd->variables.sql_mode & MODE_ONLY_FULL_GROUP_BY &&
> > >!outer_fixed &&
> > >select &&
> > > diff --git a/sql/table.h b/sql/table.h
> > > index 38b63d285c6..ab1d96538c0 100644
> > > --- a/sql/table.h
> > > +++ b/sql/table.h
> > > @@ -1374,8 +1374,13 @@ struct TABLE
> > >*/
> > >bool auto_increment_field_not_null;
> > >bool insert_or_update; /* Can be used by the handler */
> > > +  /*
> > > + NOTE: alias_name_used is only a hint! It works only in 
> > > need_correct_ident()
> > > + condition. On other cases it is FALSE even if table_name is alias!
> >
> > As I asked earlier, do you have any test case that proves this claim?
> 
> I replied that MDEV-25672 bug's own test case proves that, didn't I?
> 
> (rr) p alias_name_used
> $4 = false
> (rr) p table_name
> $5 = 0x7fdc28024790 "x"
> 
> #0  Item_field::set_field (this=0x7fdc28012e60,
> field_par=0x7fdc28024440) at /home/midenok/src/mariadb/10
> .3/src/sql/item.cc:3289
> #1  0x00b7bc28 in Item_field::fix_fields (this=0x7fdc28012e60,
> thd=0x7fdc28000d28, reference=0x7fdc28034ad8) at 
> /home/midenok/src/mariadb/10.3/src/sql/item.cc:6332
> ...
> #15 0x007c65f6 in mysql_parse (thd=0x7fdc28000d28,
> rawbuf=0x7fdc28010520 "UPDATE `test_table` as
> `x` SET order_date = NULL", length=48, parser_state=0x7fdc39d09548,
> is_com_multi=false, is_next_command=false) at 
> /home/midenok/src/mariadb/10.3/src/sql/sql_parse.cc:7870

indeed, thanks. Could you add it to the comment? Like "e.g. in update t1 as x 
set a = 1"

> > > +  */
> > >bool alias_name_used;  /* true if table_name is alias */
> > >bool get_fields_in_item_tree;  /* Signal to fix_field */
> > > +  List vcol_cleanup_list;
> > >CHARSET_INFO *save_character_set_client= 
> > > thd->variables.character_set_client;
> > >CHARSET_INFO *save_collation= thd->variables.collation_connection;
> > >Query_arena  *backup_stmt_arena_ptr= thd->stmt_arena;
> > > @@ -2831,36 +2841,162 @@ static bool fix_vcol_expr(THD *thd, 
> > > Virtual_column_info *vcol)
...
> > > +bool Vcol_expr_context::init()
> > > +{
> > > +  /*
> > > +  As this is vcol expression we must narrow down name resolution to
> > > +  single table.
> > > +  */
> > > +  if (init_lex_with_single_table(thd, table, ))
> >
> > Do you have a test that fails otherwise?
> 
> That fails at least 3 tests:
> 
> main.default vcol.vcol_syntax gcol.gcol_bugfixes
> 
> CURRENT_TEST: main.default
> mysqltest: At line 1302: query 'INSERT INTO t1 VALUES( DEFAULT )'
> failed: 2013: Lost connection to MySQL server during query

doesn't crash for me
(after building your branch and commenting out lex swapping)

> CURRENT_TEST: vcol.vcol_syntax
> mysqltest: At line 118: query 'select * from t1' failed: 2013: Lost
> connection to MySQL server during query

doesn't crash for me. I got a warning about bad double value 'x'
likely because count_cuted_fields being initialized differently in
a new lex or something.

> CURRENT_TEST: gcol.gcol_bugfixes
> mysqltest: At line 579: query 'INSERT INTO t1 (suppliersenttoday)
> VALUES (0)' failed: 2013: Lost connection to MySQL server during query

this one crashes. on the next line, 580, though.
because you set CONTEXT_ANALYSIS_ONLY_VCOL_EXPR, so
Type_std_attributes::agg_item_set_converter does not wrap items
in Item_func_conv_charset. Which is likely incorrect, because items
without wrapping cannot be properly evaluated, and it looks like they
has to be evaluated later, so it's not "context analysys only".

Crash on wrapped items is https://jira.mariadb.org/browse/MDEV-25638
that Sanja is looking at right now.

Regards,
Sergei
VP of MariaDB Server Engineering
and secur...@mariadb.org

___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net