[Maria-developers] MariaDB config for Akonadi - suggestions?

[Sergey Petrunia]

Hi,

Daniel Vratil from Akonadi (a PIM manager for KDE) has asked me at FOSDEM about
optimal MariaDB configuration for Akonadi.

Some specific questions:
* innodb_flush_log_at_trx_commit. Currently, Akonadi uses
innodb_flush_log_at_trx_commit=2 which as far as I understand can cause up to 1
second of changes to be lost. Using value of 1 (always flush on commit) should 
be safer, but maybe it would put an unnecessary load on the system? Akonadi's 
use case has a bit different idea of what load is - there is only one person 
making changes, but the system is likely to be a laptop where one cares about 
battery life, etc.

* innodb_buffer_pool_size=80M. The dataset size can vary from very small to up
to 1-1.5Gb.  Some users have very little data and complain that 80M is too 
much. 
Does anybody have an experience with using very small innodb_buffer_pool_size?

Please find the full config file below:

=== Akonadi configuration file ===
# These settings can be adjusted using $HOME/.config/akonadi/mysql-local.conf
#
# Based on advice by Kris Köhntopp 
#
[mysqld]

# strict query parsing/interpretation
# TODO: make Akonadi work with those settings enabled
# 
sql_mode=strict_trans_tables,strict_all_tables,strict_error_for_division_by_zero,no_auto_create_user,no_auto_value_on_zero,no_engine_substitution,no_zero_date,no_zero_in_date,only_full_group_by,pipes_as_concat
# sql_mode=strict_trans_tables

# DEBUGGING:
# log all queries, useful for debugging but generates an enormous amount of data
# log=mysql.full
# log queries slower than n seconds, log file name relative to datadir (for 
debugging only)
# log_slow_queries=mysql.slow
# long_query_time=1
# log queries not using indices, debug only, disable for production use
# log_queries_not_using_indexes=1
#
# mesure database size and adjust innodb_buffer_pool_size
# SELECT sum(data_length) as bla, sum(index_length) as blub FROM 
information_schema.tables WHERE table_schema not in ("mysql", 
"information_schema");

# NOTES:
# Keep Innob_log_waits and keep Innodb_buffer_pool_wait_free small (see show 
global status like "inno%", show global variables)

#expire_logs_days=3

#sync_bin_log=0

# Use UTF-8 encoding for tables
character_set_server=utf8
collation_server=utf8_general_ci

# use InnoDB for transactions and better crash recovery
default_storage_engine=innodb

# memory pool InnoDB uses to store data dictionary information and other 
internal data structures (default:1M)
# Deprecated in MySQL >= 5.6.3
innodb_additional_mem_pool_size=1M

# memory buffer InnoDB uses to cache data and indexes of its tables 
(default:128M)
# Larger values means less I/O
innodb_buffer_pool_size=80M

# Create a .ibd file for each table (default:0)
innodb_file_per_table=1

# Write out the log buffer to the log file at each commit (default:1)
innodb_flush_log_at_trx_commit=2

# Buffer size used to write to the log files on disk (default:1M for builtin, 
8M for plugin)
# larger values means less I/O
innodb_log_buffer_size=1M

# Size of each log file in a log group (default:5M) larger means less I/O but 
more time for recovery.
innodb_log_file_size=64M

# # error log file name, relative to datadir (default:hostname.err)
log_error=mysql.err

# print warnings and connection errors (default:1)
log_warnings=2

# Convert table named to lowercase
lower_case_table_names=1

# Maximum size of one packet or any generated/intermediate string. (default:1M)
max_allowed_packet=32M

# Maximum simultaneous connections allowed (default:100)
max_connections=256

# The two options below make no sense with prepared statements and/or 
transactions
# (make sense when having the same query multiple times)

# Memory allocated for caching query results (default:0 (disabled))
query_cache_size=0

# Do not cache results (default:1)
query_cache_type=0

# Do not use the privileges mechanisms
skip_grant_tables

# Do not listen for TCP/IP connections at all
skip_networking

# The number of open tables for all threads. (default:64)
table_open_cache=200

# How many threads the server should cache for reuse (default:0)
thread_cache_size=3

# wait 365d before dropping the DB connection (default:8h)
wait_timeout=31536000

# We use InnoDB, so don't let MyISAM eat up memory
key_buffer_size=16K

[client]
default-character-set=utf8

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] GROUP BY with non-grouped column returns NULL in some cases?

[Sergey Petrunia]

Hi,

On Thu, Feb 20, 2014 at 10:28:52AM -0800, Pavel Ivanov wrote:
> I've noticed a difference in MariaDB's behavior compared to its
> documentation and compared to MySQL 5.1 (didn't check with MySQL 5.5
> or 5.6 yet), and I wanted to know what's your opinion -- is it a bug
> or a conscious difference and documentation is just lagging.
> 
> This doc https://mariadb.com/kb/en/select/#group-by says: "If you
> select a non-grouped column or a value computed from a non-grouped
> column, it is undefined which row the returned value is taken from". I
> believe this implies that selected value must be returned from a row
> group, not just something random. 

I would agree with this reasoning.

MySQL 5.5, MariaDB 5.5, and MySQL 5.6 will all return (5,c,4,b).

MariaDB 10.0 returns (5,c,NULL,NULL). 

I have investigated where the NULL values come from.  They come from the next
value group. 

=== TL;DR: investigation result ==

Let's run this on 10.0 (I have added a WHERE to remove irrelevant rows):

mysql> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT OUTER JOIN t t2 ON
(t1.parent = t2.id and t2.name <> 'a') where t1.id > 4 GROUP BY t1.id;
++--+--+--+
| id | name | id   | name |
++--+--+--+
|  5 | c| NULL | NULL |
| 10 | d| NULL | NULL |
++--+--+--+

The query has GROUP BY, which is not optimized away (by either MySQL or
MariaDB). GROUP BY is satisified by using "range" access on t1, which produces
records in the order where groups follow one another.

First, end_send_group() sees the row (5,c,...). This is the first row, so it
starts a group.
Then, end_send_group() is invoked for row (10,d,...). It sees that the GROUP BY
expression value has changed, and tries to print the first group. The execution 
is here:

 #0  Field::is_null
 #1  Protocol_text::store 
 #2  Item_field::send 
 #3  Protocol::send_result_set_row 
 #4  select_send::send_data 
 #5  end_send_group 
 #6  evaluate_null_complemented_join_record 
 #7  sub_select 
 #8  evaluate_join_record 
 #9  sub_select 

The field object at #0 points to the "result field" of table t2  (a TABLE 
object has "current row" fields and "result fields". AFAIU, the latter are 
kind of buffers that are used e.g. for stashing away GROUP BY columns).

Unfortunately, Field::is_null() checks table->null_row, which is 1, because
row combination (10,d, ...) has a NULL-complemented row for t2.  This is 
why we get NULLs instead of some value from (5,c,...) group.

== Where does this come from ==

It seems, MySQL 5.6 had this bug in the past. We've got the code line where
the problem happens in 

revid: mo...@askmonty.org-20130325220313-r7oxaau0x0r5ead1, 
  Temporary commit of 10.0-merge


Since then, MySQL 5.6 has fixed it, but their related commits look scary (fix
for another fix which doesn't fully fix it?): 

revid: sergey.gluk...@oracle.com-20130619102408-0a0bf6sl5d7tsys
Bug#16620047 INCORRECT QUERY RESULT (AFTER SERVER UPGRADE)
  Regression is introduced in Bug11760517. According to the
  comment for the Bug11760517 fix, there is an additional cleanup
  in field.h(Field::is_null() function):
  Field::is_null() now checks table->null_row before checking
  the NULL bits.

  Order of NULLity checking is changed, firts it checks table->null_row
  and then null_ptr[row_offset]. In our case when we evaluate first
  result record, evaluate_null_complemented_join_record() function
  is called for outer table and it sets table->null_row to true.
  Then second result record field uses table->null_row to check
  if the field is null and result is NULL instead of real value.

  The fix is partial reversion of the field.h change
  (return original order of NULL check). Unfortunately this
  is not sufficient. A lot of test fails in the main suite
  after reversion. It happens because of another additional
  cleanup introduced Bug11760517 fix(see the comment):
 @ sql/table.h
   mark_as_null_row() no longer sets the NULL bits of the buffer

  Reversion of this change fixes all the problems except one, test case
  for Bug13541761 which is actually another regression of Bug11760517 fix.
  The fix of this problem is to not touch TABLE::null_flags during
  JOIN::cleanup.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Proposal for: MDEV-5244: Make extended_keys=ON by default in 10.0

[Sergey Petrunia]

Hello,

I was looking at how to make extended_keys=ON by default in 10.0.

There haven't been any bugs in the feature for quite some time, but we still
have a question of how to be with the testsuite (do we run it with
extended_keys=OFF, the old way, or with extended_keys=ON, the new way? both
approaches have their advantages)

So, I've just changed the default, and ran the default set of test suites.

A total 12 tests fail, of those only 3 tests fail in the way that indicates
that extended keys feature had an impact on the testcase (e.g. it got a
different execution plan and probably no longer tests what it used to test).

For these, I intend to place 

  set opimizer_switch='extended_keys=off'
  
  set opimizer_switch='extended_keys=on'

around the affected testcases. The cases are:

range_vs_index_merge_innodb - switch @@optimizer_switch (~3 testcases)
index_merge_innodb - switch @@optimizer_switch (1 testcase)
innodb.innodb_mysql -switch @@optimizer_switch (1 testcase)

Any comments/input on this?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog


___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] MySQL Optimizer Trace Questions

[Sergey Petrunia]

Hi Christian,

On Thu, Feb 20, 2014 at 06:31:45PM -0500, Christian Rober wrote:
> I work for TokuTek as a Storage Engine developer and I have been using
> the, relatively, new optimizer trace functionality.  I have found the
> results very helpful when combined with analyzing the source code, but
> there a few questions I want to ask.  Here is some output from the
> trace for a recent SELECT query on an InnoDB table:
> 
> At the end of the trace, it shows the index chosen by, what I presume,
> is the optimizer:
> 
> ...
> "considered_execution_plans":
>   [
> {
>   "plan_prefix": [],
>   "table": "`reports` `a`",
>   "best_access_path":
> {
> "considered_access_paths": [
>{
>  "access_type": "ref",
>  "index": "reports_group_id",
>  "rows": 1.55e6,
>  "cost": 411057,
>  "chosen": true
>},
>...
>  ]
>  },
>"cost_for_plan": 411057,
>"rows_for_plan": 1.55e6,
>"chosen": true
>  }
>   ]
> },
> ...
> 
> Here are the cost results for some rejected alternative indexes,
> (note: the row count for two of them is similar to the chosen index):
> 
> ...
> "analyzing_range_alternatives": {
>   "range_scan_alternatives"
> ...
>   "rows": 2377102,
>   "cost": 2.81e6,
> ...
>   "rows": 1554164,
>   "cost": 1.84e6,
> ...
>   "rows": 2346664,
>   "cost": 2.78e6,
> ...
>   "rows": 1554164,
>   "cost": 1.84e6,
> ...
> 
> Here is the cost analysis for a table scan, from the beginning of the
> trace (still not as good as the above chosen index, but close on
> cost):
> 
> ...
> "table": "`reports` `a`",
> "range_analysis":
>  {
>"table_scan":
> {
>   "rows": 3108329,
>   "cost": 655076
> },
> ...
> 
> Related to the fact that the table scan analysis is dominated by the
> large row count, and probably rejected as a valid plan compared to the
> chosen index for that very reason:
> 
> 1. How is the row count more than merely informational to the
> optimizer?  I had the impression the row count did not directly
> contribute to the optimizer's decision as much as the estimated cost,
> or that the row count was already accounted for/factored into in the
> final cost measurement.
> 2.  Related to that question, what is cost exactly?  Is there a unit
> of measurement (ex: Time, Disk Seeks, etc.)?

The cost is "random IO disk seeks". However, a lot of parts of the cost model
do not distinguish between random and sequential disk io, do not account for
presence of caches, etc.

Another important constant is (1/TIME_FOR_COMPARE). This is a cost of
evaluating the WHERE clause (or some part of it).

> 3.  Is cost more important than row count or equally important?

The decision which query plan should be used is made based on cost.

However, the join optimizer's cost calculations put a big penalty on query 
plans that produce lots of rows. See sql_select.cc, 
best_extension_by_limited_search():

...
  /* Compute the cost of extending the plan with 's' */

  current_record_count= record_count * position->records_read;
  current_read_time=read_time + position->read_time +
current_record_count / (double) TIME_FOR_COMPARE;
...

Note the last part, current_record_count/TIME_FOR_COMPARE. 
current_record_count is "# random disk reads", so adding
current_record_count/10  puts a big penalty on query plans that return a lot of
rows.

> 4.  What is the genesis of the cost variable displayed in the trace?
> I assume it is just a guess on the respective storage engine's part of
> how expensive (in terms of disk access, processing, time, etc.)
> getting the rows will be for the query.

Cost numbers for basic operations (range scan, full table scan) come from the
storage engine. Cost numbers for more complex query plans (joins,
index_merge, etc) come from cost calculations done at the SQL layer.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] MDEV-4309: Review for 201402181003_sanja_copy_calls_patch.diff

[Sergey Petrunia]

Hi Sanja,

My input for 201402181003_sanja_copy_calls_patch.diff:

* memcpy_field_possible() should have a function comment mentioning that it
  checks whether memcpy_field_value() can be used.

* field_conv_incompatible() should have a function comment mentioning that 
  it is a function that is used instead of field_conv() when we know in 
  advance that memcpy_field_possible(...)==false.

Ok to push after this is addressed.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Engine independent index statistics - infrastructure questions

[Sergey Petrunia]

Hello Igor and others,

I've been playing around with engine independent statistics, 
and got a few questions.

== mysqldump ==
mysqldump dumps table' data, but besides that it can dump other related
objects, e.g. it will dump triggers that are attached to the table.
Should it dump table statistics values, also? 

This question is of interest for DBAs, but also for optimizer developers. If
a user has a complaint about the optimizer, we frequently ask them to upload a
dataset. How do we request table/column statistics? We can't request output of 
SELECT * ..., because histogram data is binary and will be corrupted.  We
should provide users a convinent way to pass us relevant part of the EITS
tables.

== dropping table statistics ==
How does one drop table statistics? are we satisfied with needing to run
three statements

DELETE FROM mysql.column_stats WHERE schema_name=$db AND table_name=$table_name;
DELETE FROM mysql.index_stats  WHERE table_name=$db AND table_name=$table_name;
DELETE FROM mysql.column_stats WHERE table_name=$db AND table_name=$table_name;

or we want a simpler SQL, like

DROP STATISTICS FOR t1;

== Permissions on statistic tables ==

What should be the permissions on statistics tables?  I guess we should follow
pattern from other system tables - a user is typically able to read the table
but not write to it (so that he can't mess with other users' SPs/triggers/etc).

However, if the user is not allowed to modify the table directly, we should
provide enough ways to modify the table indirectly. For example, we should
provide a way to drop statistics for one's tables (which brings us back to the
previous item). We might also want to allow to make other modifications to
statistics of tables that one has control of?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Engine independent statistics - naming issues?

[Sergey Petrunia]

Hello,

I was looking at whether column names used by EIS feature are consistent with
themselves and with ther parts of the server.

== Database / table / column names ==

There are a few naming conventions arready in use:

1. Old school:
mysql.columns_priv, tables_priv
  Db
  Table_name
  Column_name

2. SQL STANDARD ALL CAPS HELLO FROM 1970S:
information_schema.*  (tables, columns, indexes, index_statistics, etc)
  TABLE_CATALOG (always 'def')
  TABLE_SCHEMA
  TABLE_NAME
  COLUMN_NAME
  INDEX_NAME

3. InnoDB has its own convention:
mysql.innodb_table_stats, innodb_index_stats
  database_name
  table_name
  index_name

4. EITS feature will add:
mysql.column_stats, index_stats, table_stats
  db_name
  table_name
  column_name
  index_name

== Other ==

There are variables @@histogram_size, @@histogram_type.
mysql.column_stats has columns named hist_size, hist_type.


What's your opinion, should we make some renames to make things more
consistent?
 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] 10.2 tree, innodb_gis.alter_spatial_index test broken after your push

[Sergey Petrunia]

Hi Jan,

Your push
https://github.com/MariaDB/server/commit/dc9f919f27fccfeb0de3ab392f33bc5efdfd59a0

broke the innodb_gis.alter_spatial_index test in 10.2 tree.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] MariaRocks port: why handler::init_with_fields() call is needed + call for comments

[Sergey Petrunia]

Hello,

I think me and Serg have tried having this discussion before on several
occasions but were not successful due to me not having the details back then.
Now I'm trying with the details.

== Description of the issue ==
Look at MariaDB, the code in TABLE_SHARE::init_from_binary_frm_image (or
open_binary_frm() in mysql-5.6 codebase)

The code goes like this:

  ...

 /* Allocate handler */
  if (!(handler_file= get_new_handler(share, thd->mem_root,
  plugin_hton(se_plugin
  // ^(LOCATION-1)
  ...

  /* Fix key->name and key_part->field */
  if (key_parts)
  {
longlong ha_option= handler_file->ha_table_flags(); 
// ^(LOCATION-2)
...

for (uint key=0 ; key < keys ; key++,keyinfo++)
{
  ...
  for (i=0; i < key_parts; key_part++, i++)
  {
...
field= key_part->field= share->field[key_part->fieldnr-1];
// (LOCATION-3)
...
  }
}

Note that
- a handler object is created at (LOCATION-1). A TABLE_SHARE object exists
at that point, but it has table_share->key_info==NULL. This means the storage
object has no information about the indexes in the table it is representing.

- At (LOCATION-2), the code retrieves ha_table_flags(). A bit below that point
it checks for "ha_option & HA_PRIMARY_KEY_IN_READ_INDEX" (there are also other
checks, both harmful and harmless, but let's focus on this one for now).

In MyRocks, whether HA_PRIMARY_KEY_IN_READ_INDEX flag is set depends on the 
datatypes of the primary key columns. However, ha_rocksdb object is requested 
to provide ha_table_flags() before it has a chance of looking at the table 
DDL and actually knowing the value.

== Solution in MyRocks ==
It's here:
https://github.com/facebook/mysql-5.6/commit/c1f1f0b64eaecd132fc6ec21663faf0510afacaf

Basically, there is a new handler::init_with_fields() call which SQL layer 
calls after information about table fields is filled.

== Questions ==

Any objections to this approach? 
Alternative solutions that do not require a lot of re-writing?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Review for version#2 of MDEV-11297: Add support for LIMIT clause in GROUP_CONCAT

[Sergey Petrunia]

Hi Varun,

Nice to see that nearly all comments from the previous review were addressed.
Please find my further input below.

General questions:
How is DISTINCT handled? Syntax like

  GROUP_CONCAT(DISTINCT col ORDER BY ... LIMIT ...)

is allowed by the parser, however I don't see any handling for DISTINCT in your
patch.

Did you forget to update group_concat_limit.result file? Please check.


> diff -urpN '--exclude=.*' 10.2-varun-mdev11297-orig/sql/item_sum.cc 
> 10.2-varun-mdev11297-noc/sql/item_sum.cc
> --- 10.2-varun-mdev11297-orig/sql/item_sum.cc 2016-12-11 12:17:04.023961449 
> +0300
> +++ 10.2-varun-mdev11297-noc/sql/item_sum.cc  2016-12-11 12:18:40.411957365 
> +0300
> @@ -3114,6 +3114,11 @@ int dump_leaf_key(void* key_arg, element
>String *result= &item->result;
>Item **arg= item->args, **arg_end= item->args + item->arg_count_field;
>uint old_length= result->length();
> +  ulonglong *offset_limit= &item->copy_offset_limit;
> +  ulonglong *row_limit = &item->copy_row_limit;
> +
> +  if (item->limit_clause && !(*row_limit))
> + return 1;
>  
>if (item->no_appended)
>  item->no_appended= FALSE;
> @@ -3122,20 +3127,23 @@ int dump_leaf_key(void* key_arg, element
>  
>tmp.length(0);
>  
> -  for (; arg < arg_end; arg++)
> +  if (!item->limit_clause || !(*offset_limit))
>{
> -String *res;
> -/*
> -  We have to use get_tmp_table_field() instead of
> -  real_item()->get_tmp_table_field() because we want the field in
> -  the temporary table, not the original field
> -  We also can't use table->field array to access the fields
> -  because it contains both order and arg list fields.
> - */
> -if ((*arg)->const_item())
> -  res= (*arg)->val_str(&tmp);
> -else
> +
> +for (; arg < arg_end; arg++)
>  {
> +  String *res;
> +  /*
> +We have to use get_tmp_table_field() instead of
> +real_item()->get_tmp_table_field() because we want the field in
> +the temporary table, not the original field
> +We also can't use table->field array to access the fields
> +because it contains both order and arg list fields.
> +  */
> +  if ((*arg)->const_item())
> +res= (*arg)->val_str(&tmp);
> +  else
> +  {
>Field *field= (*arg)->get_tmp_table_field();

Please make the code to use the right indentation.

>if (field)
>{
> @@ -3146,9 +3154,19 @@ int dump_leaf_key(void* key_arg, element
>}
>else
>  res= (*arg)->val_str(&tmp);
> +  }
> +  if (res)
> +  {
> +result->append(*res);
> +  }
>  }
> -if (res)
> -  result->append(*res);
> +if ((*row_limit))
> +(*row_limit)--;
> +  }
> +  else
> +  {
> +item->no_appended= TRUE;

If I comment out the above line, the test still passes. Please add test
coverage that shows it is needed, or remove it.

> +(*offset_limit)--;
>}
>  
>item->row_count++;
> @@ -3199,7 +3217,8 @@ Item_func_group_concat::
>  Item_func_group_concat(THD *thd, Name_resolution_context *context_arg,
> bool distinct_arg, List *select_list,
> const SQL_I_List &order_list,
> -   String *separator_arg)
> +   String *separator_arg, bool limit_clause,
> +   Item *rowLimit, Item *offsetLimit)

camelCase or javaStyleVariableNames are against the coding style.
Please use something like row_limit_arg if there is no better option.

(In general, please try to make your new code fit in with the rest of the code.
Did you see camelCase anywhere? If not, it's a sign not to use it).

>:Item_sum(thd), tmp_table_param(0), separator(separator_arg), tree(0),
> unique_filter(NULL), table(0),
> order(0), context(context_arg),
> @@ -3208,7 +3227,9 @@ Item_func_group_concat(THD *thd, Name_re
> row_count(0),
> distinct(distinct_arg),
> warning_for_row(FALSE),
> -   force_copy_fields(0), original(0)
> +   force_copy_fields(0),row_limit(0),
> +   offset_limit(0),limit_clause(limit_clause),
> +   copy_offset_limit(0), copy_row_limit(0),original(0)

Please use 'NULL' for pointers, not '0'.

>  {
>Item *item_select;
>Item **arg_ptr;
> @@ -3250,6 +3271,16 @@ Item_func_group_concat(THD *thd, Name_re
>/* orig_args is only used for print() */
>orig_args= (Item**) (order + arg_count_order);
>memcpy(orig_args, args, sizeof(Item*) * arg_count);
> +
> +  if (limit_clause)
> +  {
> +if (!(row_limit=(Item*) thd->alloc(sizeof(Item
> +  return;

I'm not sure what the above is supposed to mean? Why allocate sizeof(Item)
bytes?

> +row_limit= rowLimit;

... especially when the next thing you do is the above assignment, so you 
lose the value that you've got from the thd->alloc() call?

> +if (!(offset_limit= (Item*)thd->alloc(sizeof(Item
> +  return;
> +offset_limit= offsetLimit;

The same question as above.

> +  }
>  }
>  

Re: [Maria-developers] [Commits] 9bd354e: MDEV-11060 sql/protocol.cc:532: void Protocol::end_statement(): Assertion `0' failed

[Sergey Petrunia]

Hi Varun,

Please find a small note below. Ok to push after it has been addressed.

On Tue, Dec 13, 2016 at 05:09:58AM +0530, Varun wrote:
> revision-id: 9bd354ef91ae81e0f997f29acdae4f8a2997b3c8 
> (mariadb-10.1.19-40-g9bd354e)
> parent(s): 67b570af506a42f14a8ae30603d52ca9de68cce5
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2016-12-13 05:07:02 +0530
> message:
> 
> MDEV-11060 sql/protocol.cc:532: void Protocol::end_statement(): Assertion `0' 
> failed
> 
> In file sql/opt_range.cc,when calculate_cond_selectivity_for_table() is 
> called with optimizer_use_condition_selectivity=4 then
>   - thd->no_errors is set to 1
>   - the original value of thd->no_error is not restored to its original 
> value
>   - this is causing the assertion to fail in the subsequent queries
> 
> Fixed by restoring the original value of thd->no_errors
> 
> ---
>  mysql-test/r/selectivity_innodb.result | 23 +++
>  mysql-test/t/selectivity_innodb.test   | 28 
>  sql/opt_range.cc   |  1 +
>  3 files changed, 52 insertions(+)
> 
> diff --git a/mysql-test/r/selectivity_innodb.result 
> b/mysql-test/r/selectivity_innodb.result
> index 4355812..b57873d 100644
> --- a/mysql-test/r/selectivity_innodb.result
> +++ b/mysql-test/r/selectivity_innodb.result
> @@ -1670,6 +1670,29 @@ DROP TABLE t1,t2;
>  #
>  # End of 10.0 tests
>  #
> +#
> +# MDEV-11060: sql/protocol.cc:532: void Protocol::end_statement(): Assertion 
> `0' failed
> +#
> +set optimizer_use_condition_selectivity=4;
> +drop table if exists t1, t2;
> +Warnings:
> +Note 1051Unknown table 'test.t1'
> +Note 1051Unknown table 'test.t2'
> +drop view if exists v1;
> +Warnings:
> +Note 1051Unknown table 'test.v1'

Please remove 'DROP TABLE IF EXISTS' statements as they are not necessary.
The previous test ends with DROP TABLE commands for tables t1, t2.

There is no DROP VIEW command, so it is useful that the VIEW is dropped before
it is used (just in case the test is run against a server that already has the
view).
Check out other test files, most of them start with lines like:

--disable_warnings
drop table if exists t1,t2;
--enable_warnings

Please also suppress warnings for the DROP VIEW statement.

> +create table t1 (a int not null, b int, c int) engine=InnoDB;
> +create trigger trgi before insert on t1 for each row set new.a=if(new.a is 
> null,new.b,new.c);
> +create table t2 (d int, e int) engine=InnoDB;
> +update t1, t2 set a=NULL, b=2, c=NULL where b=d and e=200;
> +create view v1 as select * from t1, t2 where d=2;
> +insert v1 (a,c) values (NULL, 20);
> +ERROR 23000: Column 'a' cannot be null
> +drop table t1,t2;
> +drop view v1;
> +#
> +# End of 10.1 tests
> +#
>  set use_stat_tables= @tmp_ust;
>  set optimizer_use_condition_selectivity= @tmp_oucs;
>  SET SESSION STORAGE_ENGINE=DEFAULT;
> diff --git a/mysql-test/t/selectivity_innodb.test 
> b/mysql-test/t/selectivity_innodb.test
> index 25aa0ab..bb87106 100644
> --- a/mysql-test/t/selectivity_innodb.test
> +++ b/mysql-test/t/selectivity_innodb.test
> @@ -138,6 +138,34 @@ DROP TABLE t1,t2;
>  --echo # End of 10.0 tests
>  --echo #
>  
> +
> +--echo #
> +--echo # MDEV-11060: sql/protocol.cc:532: void Protocol::end_statement(): 
> Assertion `0' failed
> +--echo #
> +
> +
> +set optimizer_use_condition_selectivity=4;
> +
> +drop table if exists t1, t2;
> +drop view if exists v1;
> +
> +create table t1 (a int not null, b int, c int) engine=InnoDB;
> +create trigger trgi before insert on t1 for each row set new.a=if(new.a is 
> null,new.b,new.c);
> +
> +create table t2 (d int, e int) engine=InnoDB;
> +update t1, t2 set a=NULL, b=2, c=NULL where b=d and e=200;
> +
> +create view v1 as select * from t1, t2 where d=2;
> +--error ER_BAD_NULL_ERROR
> +insert v1 (a,c) values (NULL, 20);
> +
> +drop table t1,t2;
> +drop view v1;
> +
> +--echo #
> +--echo # End of 10.1 tests
> +--echo #
> +
>  set use_stat_tables= @tmp_ust;
>  set optimizer_use_condition_selectivity= @tmp_oucs;
>  SET SESSION STORAGE_ENGINE=DEFAULT;
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index bed57f3..0577ef0 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -3112,6 +3112,7 @@ bool calculate_cond_selectivity_for_table(THD *thd, 
> TABLE *table, Item **cond)
>  }
>  
>free_alloc:
> +thd->no_errors= 0;
>  thd->mem_root= param.old_root;
>  free_root(&alloc, MYF(0));
>  
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 12034ae: MDEV-10927: Crash When Using sort_union Optimization

[Sergey Petrunia]

The patch is ok, but there is a problem with the testcase: when I run the
testcase on the unmodified 5.5 binary, it passes.

When I run the tescase from the MDEV, I get the crash.
Please fix the testcase. This will also provide test coverage (which was
lacking, and this is why have this bug in the first place).

On Fri, Dec 16, 2016 at 11:19:43AM +0530, Varun wrote:
> revision-id: 12034aec831d776725a43c7537ab4256efb1c974 
> (mariadb-5.5.53-25-g12034ae)
> parent(s): 14e1f32894cdbe63a614738cfd93e9d0818dedee
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2016-12-16 11:03:24 +0530
> message:
> 
> MDEV-10927: Crash When Using sort_union Optimization
> 
> In file sql/filesort.cc,when merge_buffers() is called then
> - queue_remove(&queue,0) is called
> - For the function queue_remove there is assertion states that the element to 
> be removed should have index >=1
> - this is causing the assertion to fail.
> 
> Fixed by removing the top element.
> 
> ---
>  mysql-test/r/index_merge_innodb.result | 52 
> ++
>  mysql-test/t/index_merge_innodb.test   | 42 ++-
>  sql/filesort.cc|  2 +-
>  3 files changed, 94 insertions(+), 2 deletions(-)
> 
> diff --git a/mysql-test/r/index_merge_innodb.result 
> b/mysql-test/r/index_merge_innodb.result
> index b93d15f..49e0a57 100644
> --- a/mysql-test/r/index_merge_innodb.result
> +++ b/mysql-test/r/index_merge_innodb.result
> @@ -792,4 +792,56 @@ ab   c
>  8m   m
>  9d   d
>  DROP TABLE t1;
> +#
> +# MDEV-10927: Crash When Using sort_union Optimization
> +#
> +SET optimizer_switch='index_merge_sort_intersection=on';
> +CREATE TABLE IF NOT EXISTS t1 (
> +id int(11) NOT NULL AUTO_INCREMENT,
> +contact_id int(11) NOT NULL,
> +role_id int(11) NOT NULL,
> +company_type_id int(11) NOT NULL,
> +company_id int(11) NOT NULL,
> +team_id int(11) NOT NULL,
> +user_id int(11) NOT NULL,
> +PRIMARY KEY (id),
> +KEY contact_id (contact_id),
> +KEY user_id (user_id),
> +KEY company_id (company_id),
> +KEY team_id (team_id) USING BTREE
> +) ENGINE=InnoDB AUTO_INCREMENT=12860259 DEFAULT CHARSET=utf8 
> ROW_FORMAT=COMPACT;
> +INSERT INTO t1 (id, contact_id, role_id, company_type_id, company_id, 
> team_id, user_id) VALUES
> +(5343408, 23913609, 0, 6, 2321, 0, 0),
> +(7841449, 39582328, 1194, 0, 0, 0, 57291),
> +(7841507, 28707292, 1194, 0, 0, 0, 57291),
> +(7841543, 26376778, 1194, 0, 0, 0, 137571),
> +(7841563, 42475686, 1194, 0, 0, 0, 137571),
> +(7841573, 35956831, 1194, 0, 0, 0, 447223),
> +(7841575, 37975231, 1194, 0, 0, 0, 447223),
> +(7843049, 42693923, 1194, 0, 0, 0, 376261),
> +(7843600, 42696877, 1194, 0, 0, 0, 376261),
> +(12858344, 82851915, 1194, 0, 0, 0, 993011),
> +(12858554, 80183799, 1194, 0, 0, 0, 993011),
> +(12858803, 82649193, 1194, 0, 0, 0, 1288749),
> +(12858878, 82700925, 1194, 0, 0, 0, 1288749),
> +(12860248, 28642729, 1194, 0, 0, 0, 1234288),
> +(12860258, 28642729, 0, 0, 0, 0, 1234288),
> +(12860259, 28642729, 1194, 0, 0, 0, 0);
> +SELECT SQL_NO_CACHE contact_id FROM t1 FORCE INDEX (user_id,company_id) 
> WHERE user_id = '57291' OR (user_id 
> IN(55838,55839,56998,57260,57291,60227,121997,137571,173600,219233,306480,354991,358594,376261,398572,447223,472009,646282,932548,993011,1004609,1021262,1177117,1234288,1283121,1288749,1337771,57291)
>  AND role_id = '1194') OR company_id = '2321';
> +contact_id
> +23913609
> +26376778
> +28642729
> +28707292
> +35956831
> +37975231
> +39582328
> +42475686
> +42693923
> +42696877
> +80183799
> +82649193
> +82700925
> +82851915
> +DROP TABLE t1;
>  set optimizer_switch= @optimizer_switch_save;
> diff --git a/mysql-test/t/index_merge_innodb.test 
> b/mysql-test/t/index_merge_innodb.test
> index 6a1cb53..f1fdbbd 100644
> --- a/mysql-test/t/index_merge_innodb.test
> +++ b/mysql-test/t/index_merge_innodb.test
> @@ -172,5 +172,45 @@ WHERE ( tb.b != ta.b OR tb.a = ta.a )
>  
>  DROP TABLE t1;
>  
> -set optimizer_switch= @optimizer_switch_save;
> +--echo #
> +--echo # MDEV-10927: Crash When Using sort_union Optimization
> +--echo #
>  
> +SET optimizer_switch='index_merge_sort_intersection=on';
> +
> +CREATE TABLE IF NOT EXISTS t1 (
> +id int(11) NOT NULL AUTO_INCREMENT,
> +contact_id int(11) NOT NULL,
> +role_id int(11) NOT NULL,
> +company_type_id int(11) NOT NULL,
> +company_id int(11) NOT NULL,
> +team_id int(11) NOT NULL,
> +user_id int(11) NOT NULL,
> +PRIMARY KEY (id),
> +KEY contact_id (contact_id),
> +KEY user_id (user_id),
> +KEY company_id (company_id),
> +KEY team_id (team_id) USING BTREE
> +) ENGINE=InnoDB AUTO_INCREMENT=12860259 DEFAULT CHARSET=utf8 
> ROW_FORMAT=COMPACT;
> +
> +INSERT INTO t1 (id, contact_id, role_id, company_type_id, company_id, 
> team_id, user_id) VALUES
> +  (5343408, 23913609, 0, 6, 2321, 0, 0),
> +  (7841449, 39582328, 1194, 0, 0, 0, 57291),
> +  (7841507, 28707292, 1194, 0, 0, 0, 57291),
> +  (7841543, 26376778, 1194, 0, 0, 0, 13757

Re: [Maria-developers] 10.3 branches

[Sergey Petrunia]

On Mon, Dec 19, 2016 at 10:49:42AM +0400, Alexander Barkov wrote:
> Hi Sergei,
> 
> Thanks! Please see my replies inline:
> 
> On 12/16/2016 05:38 PM, Sergei Golubchik wrote:
> > Hi, Alexander!
> > 
> > Sure, 10.2-ext or 10.3-base - whatever you prefer.
> > I understand that bb-10.2-compatibility will be eventually
> > pushed into 10.3 too, when it'll be ready.
> 
> Correct.
> 
> > 
> > For now, I'd suggest the following:
> > 
> > 10.2--->10.3
> >  \
> >   \- 10.2-ext ---> 10.2-compat
> >
> > and eventually 10.2-ext and 10.2-compat are getting merged or rebased
> > into 10.3.
> 
> I created a new branch bb-10.2-ext.
> 
> So the tentative data flow looks like this:
> 
> 10.2-->10.3 <---(once)
>  \ ^  ^
>   \|  |
>\- bb-10.2-ext ---> bb-10.2-compatibility
> 
> 
> 
> - 10.2 will be periodically merged to 10.3.
> - bb-10.2-ext will be periodically rebased on top of 10.2
> - bb-10.2-compatibility will be periodically rebased on top of  bb-10.2-ext.
> - bb-10.2-ext will be periodically merged to 10.3
> - bb-10.2-compatibility will once (when it's ready) be
>   merged or rebased to 10.3
> 
> 
> Every time I rebase bb-10.2-ext on top of the current 10.2,
> I can merge bb-10.2-ext to 10.3 immediately, so we don't have to
> do double work.

Maybe I am missing something, but suppose somebodty has pushed 
the code for MDEV-10141 (for example, I am using this MDEV as it is mentioned
below) into bb-10.2-ext.

Then, you rebase bb-10.2-ext on top of 10.2.  This will cause another 
commit for MDEV-10141 to be created.
Then you merge it to 10.3. This means 10.3 will have two commits for
MDEV-10141.
If you do the above process N times, 10.3 will have 1..N copies of everything 
that was pushed into bb-10.2-ext.  Is that the intent?

> 
> 
> > 
> > Other features that depend on your refactoring and need be in the compat
> > branch, can be pushed directly into 10.2-ext.
> 
> Correct.
> 
> Note, by refactoring I mean two things:
> - Moving pieces of the code from sql_yacc.yy as methods
>   to LEX, sp_head, THD, etc.
>   This is to avoid duplicate code in sql_yacc_ora.yy.
> 
> - Some Type_handler related patches.
> 
> > 
> > Are there features that should not be in 10.2-compat but still depend on
> > your refactorings?
> 
> Sanja and I are thinking of pushing
> 
> MDEV-10141 Add support for INTERSECT (and common parts for EXCEPT)
> 
> into bb-10.2-ext when it's ready.
> 
> There will be more tasks, I think.
> 
> 
> > 
> > On Dec 16, Alexander Barkov wrote:
> >> Hello Serg, all,
> >>
> >> A few weeks ago I created a new branch "10.3" and pushed
> >> essential patches needed for the compatibility project.
> >>
> >> It worked as follows:
> >> - In 10.3:
> >>   git rebase origin/10.2
> >>   git push --force
> >>
> >> - In bb-10.2-compatibility:
> >>   git rebase origin/10.3
> >>   git push --force
> >>
> >>
> >> Now as other developers start to push into 10.3, this won't work
> >> anymore. We briefly discussed with Monty that, as the compatibility
> >> project should be as stable as possible and merging all 10.3 patches
> >> into bb-10.2-compatibility is not desirable, we could try something
> >> like this:
> >>
> >> Add a new branch (Monty proposed "10.3-base" as the name)
> >> and propagate changes as follows:
> >>
> >> - In 10.3-base:
> >>   git rebase origin/10.2
> >>   git push --force
> >>
> >> - In bb-10.2-compatibility
> >>   git rebase origin/10.3-base
> >>   git push --force
> >>
> >> - In 10.3:
> >>   git merge origin/10.3-base
> >>   git push  (no --force)
> >>
> >>
> >> 10.2 ---(rebase)--->10.3-base--(rebase)-->bb-10.2-compatibility
> >>   |   |
> >> (merge)(merge)
> >>   |   |
> >>   V   |
> >> 10.3<-/
> >>
> >>
> >> Any objections about this scheme?
> >>
> >> I'd only propose to use "10.2-ext" instead of "10.3-base",
> >> as this intermediate branch is going to be more 10.2 than 10.3.
> > 
> > Regards,
> > Sergei
> > Chief Architect MariaDB
> > and secur...@mariadb.org
> > 
> 
> ___
> Mailing list: https://launchpad.net/~maria-developers
> Post to : maria-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~maria-developers
> More help   : https://help.launchpad.net/ListHelp

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 905aac9: MDEV-10232: Scalar result of subquery changes after adding an outer select stmt

[Sergey Petrunia]

Hi Varun,


On Fri, Jan 13, 2017 at 08:22:49PM +0530, Varun wrote:
> revision-id: 905aac9c78068224839db551a84ed080167d1657 
> (mariadb-10.1.20-37-g905aac9)
> parent(s): ebb8c9fb26f86cff8c0d81bd2415f415cef952bb
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-01-13 20:17:33 +0530
> message:
> 
> MDEV-10232: Scalar result of subquery changes after adding an outer select 
> stmt
> 
> In a subquery, we don't have to maintain order
> Added a fix such that order is considered when there is a limit clause.
> 
> ---
>  mysql-test/r/subselect4.result | 12 
>  mysql-test/t/subselect4.test   | 10 ++
>  sql/sql_select.cc  |  2 +-
>  3 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/mysql-test/r/subselect4.result b/mysql-test/r/subselect4.result
> index 87645d1..7764783 100644
> --- a/mysql-test/r/subselect4.result
> +++ b/mysql-test/r/subselect4.result
> @@ -2401,5 +2401,17 @@ SELECT x FROM t1 WHERE id > (SELECT MAX(id) - 1000 
> FROM t1) ORDER BY x LIMIT 1;
>  x
>  0
>  drop table t1;
> +#
> +# MDEV-10232 Scalar result of subquery changes after adding an outer select 
> stmt
> +#
> +create table t1(c1 int, c2 int, primary key(c2));
> +insert into t1 values(2,1),(1,2);
> +select (select c1 from t1 group by c1,c2 order by c1 limit 1) as x;
> +x
> +1
> +(select c1 from t1 group by c1,c2 order by c1 limit 1);
> +c1
> +1
> +drop table t1;
>  SET optimizer_switch= @@global.optimizer_switch;
>  set @@tmp_table_size= @@global.tmp_table_size;
> diff --git a/mysql-test/t/subselect4.test b/mysql-test/t/subselect4.test
> index 4eb9701..787c579 100644
> --- a/mysql-test/t/subselect4.test
> +++ b/mysql-test/t/subselect4.test
> @@ -1956,5 +1956,15 @@ SELECT x FROM t1 WHERE id > (SELECT MAX(id) - 1000 
> FROM t1) ORDER BY x LIMIT 1;
>  
>  drop table t1;
>  
> +--echo #
> +--echo # MDEV-10232 Scalar result of subquery changes after adding an outer 
> select stmt
> +--echo #
> +
> +create table t1(c1 int, c2 int, primary key(c2));
> +insert into t1 values(2,1),(1,2);
> +select (select c1 from t1 group by c1,c2 order by c1 limit 1) as x;
> +(select c1 from t1 group by c1,c2 order by c1 limit 1);
> +drop table t1;
> +
>  SET optimizer_switch= @@global.optimizer_switch;
>  set @@tmp_table_size= @@global.tmp_table_size;
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 3f06ec8..13eda46 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -1626,7 +1626,7 @@ JOIN::optimize_inner()

Above this change we still have a comment with this statement:
>>- if we are in a subquery, we don't have to maintain order

Please adjust it accordingly.


>if (!order || test_if_subpart(group_list, order))
>{
>  if (skip_sort_order ||
> -select_lex->master_unit()->item) // This is a subquery
> +select_lex->master_unit()->item && !select_limit) // This is a 
> subquery
>order= NULL;
>  else
>order= group_list;

So I'm trying with the patch:

MariaDB [test]> explain select (select c1 from t1 group by c1,c2 order by c1 
limit 1) as x;
+--+-+---+--+---+--+-+--+--++
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | 
rows | Extra  |
+--+-+---+--+---+--+-+--+--++
|1 | PRIMARY | NULL  | NULL | NULL  | NULL | NULL| NULL | 
NULL | No tables used |
|2 | SUBQUERY| t1| ALL  | NULL  | NULL | NULL| NULL |   
 2 | Using filesort |
+--+-+---+--+---+--+-+--+--++
2 rows in set (20.08 sec)

Good.
What if there is no LIMIT clause?

MariaDB [test]> explain select (select c1 from t1 group by c1,c2 order by c1 ) 
as x;
+--+-+---+--+---+--+-+--+--++
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | 
rows | Extra  |
+--+-+---+--+---+--+-+--+--++
|1 | PRIMARY | NULL  | NULL | NULL  | NULL | NULL| NULL | 
NULL | No tables used |
|2 | SUBQUERY| t1| ALL  | NULL  | NULL | NULL| NULL |   
 2 | Using filesort |
+--+-+---+--+---+--+-+--+--++
2 rows in set (6.06 sec)

Bad. "Using filesort" should have been removed.

The reason it fails is that "!select_limit" is not the right way to check the
presense of LIMIT clause.
Without limit, I have:

(gdb) p/x select_limit
  $15 = 0x

(the name of the constant to use is "HA_POS_ERROR")


Another issue: I get this warning with the new code:

|| /home/psergey/dev-git/10.1/sql/sql_select.cc: In member function 'int 
JOIN::optimize_inner()':
sql_select.cc|1629 col 45| warning: suggest pare

Re: [Maria-developers] [Commits] eb8d31d: MDEV-10232: Scalar result of subquery changes after adding an outer select stmt

[Sergey Petrunia]

Hi Varun,

Please make the fix for MDEV-10232 to be one changeset. Ok to push that
changeset.

On Sun, Jan 15, 2017 at 07:44:35PM +0530, Varun wrote:
> revision-id: eb8d31d96f32da53d34d4c992fba061fc8732078 
> (mariadb-10.1.20-38-geb8d31d)
> parent(s): 905aac9c78068224839db551a84ed080167d1657
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-01-15 19:40:46 +0530
> message:
> 
> MDEV-10232: Scalar result of subquery changes after adding an outer select 
> stmt
> 
> postreview
> 
> ---
>  sql/sql_select.cc | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 13eda46..85bfb6b 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -1614,7 +1614,8 @@ JOIN::optimize_inner()
>   to ORDER BY . There are three exceptions:
>  - if skip_sort_order is set (see above), then we can simply skip
>GROUP BY;
> -- if we are in a subquery, we don't have to maintain order
> +- if we are in a subquery, we don't have to maintain order unless 
> there
> +   is a limit clause in the subquery.
>  - we can only rewrite ORDER BY if the ORDER BY fields are 
> 'compatible'
>with the GROUP BY ones, i.e. either one is a prefix of another.
>We only check if the ORDER BY is a prefix of GROUP BY. In this case
> @@ -1626,7 +1627,7 @@ JOIN::optimize_inner()
>if (!order || test_if_subpart(group_list, order))
>{
>  if (skip_sort_order ||
> -select_lex->master_unit()->item && !select_limit) // This is a 
> subquery
> +(select_lex->master_unit()->item && select_limit == 
> HA_POS_ERROR)) // This is a subquery
>order= NULL;
>  else
>order= group_list;
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 14747e4: MDEV-10731: Wrong NULL match results in "Subquery returns more than 1 row" (error code 1242)

[Sergey Petrunia]

Hi Varun,

On Tue, Jan 17, 2017 at 03:24:39PM +0530, Varun wrote:
> revision-id: 14747e4b84f68ee7e10a6e5a24f4b7ec6f3240f9 
> (mariadb-10.1.20-38-g14747e4)
> parent(s): b56f726e42e8bc0427a0c33cfffb7e1c5399ea16
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-01-17 15:22:22 +0530
> message:
> 
> MDEV-10731: Wrong NULL match results in "Subquery returns more than 1 row" 
> (error code 1242)
> 
> NOT NULL predicate was not added to tables in case of an update query having 
> a subquery.
> 
> ---
>  mysql-test/r/update_innodb.result | 30 +
>  mysql-test/t/update_innodb.test   | 31 +
>  sql/sql_select.cc | 70 
> +--
>  3 files changed, 106 insertions(+), 25 deletions(-)
> 
> diff --git a/mysql-test/r/update_innodb.result 
> b/mysql-test/r/update_innodb.result
> index 88c86c5..6a02dfd 100644
> --- a/mysql-test/r/update_innodb.result
> +++ b/mysql-test/r/update_innodb.result
> @@ -29,3 +29,33 @@ CREATE ALGORITHM=UNDEFINED VIEW `v1` AS select `t4`.`c1` 
> AS `c1`,`t4`.`c2` AS `c
>  UPDATE t1 a JOIN t2 b ON a.c1 = b.c1 JOIN v1 vw ON b.c2 = vw.c1 JOIN t3 del 
> ON vw.c2 = del.c2 SET a.c2 = ( SELECT max(t.c1) FROM t3 t, v1 i WHERE del.c2 
> = t.c2 AND vw.c3 = i.c3 AND t.c3 = 4 ) WHERE a.c2 IS NULL OR a.c2 < 
> '2011-05-01';
>  drop view v1;
>  drop table t1,t2,t3,t4;
> +#
> +# MDEV-10232 Scalar result of subquery changes after adding an outer select 
> stmt
> +#
The MDEV number and title are wrong! Please fix.

> +CREATE TABLE a (
Please make tables names to follow the t1,t2,t3,... naming convention used in
the testcase.

> +a_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +b_id INT(20) UNSIGNED NULL DEFAULT NULL,
> +c_id VARCHAR(255) NULL DEFAULT NULL,
> +PRIMARY KEY (a_id)
> +)
> +COLLATE = 'utf8_general_ci'
> +ENGINE = InnoDB;
> +CREATE TABLE b (
> +b_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +c_id VARCHAR(255) NULL DEFAULT NULL,
> +PRIMARY KEY (b_id),
> +INDEX idx_c_id (c_id)
> +)
> +COLLATE = 'utf8_general_ci'
> +ENGINE = InnoDB;
> +INSERT INTO a (b_id, c_id) VALUES (1, NULL);
> +INSERT INTO b (c_id) VALUES (NULL);
> +INSERT INTO b (c_id) VALUES (NULL);
> +select b_id from a;
> +b_id
> +1
> +UPDATE a SET b_id = (SELECT b.b_id FROM b b WHERE b.c_id = a.c_id);
> +select b_id from a;
> +b_id
> +NULL
> +drop table a,b;
> diff --git a/mysql-test/t/update_innodb.test b/mysql-test/t/update_innodb.test
> index 67c356c..f2b7ade 100644
> --- a/mysql-test/t/update_innodb.test
> +++ b/mysql-test/t/update_innodb.test
> @@ -37,3 +37,34 @@ UPDATE t1 a JOIN t2 b ON a.c1 = b.c1 JOIN v1 vw ON b.c2 = 
> vw.c1 JOIN t3 del ON v
>  
>  drop view v1;
>  drop table t1,t2,t3,t4;
> +
> +--echo #
> +--echo # MDEV-10232 Scalar result of subquery changes after adding an outer 
> select stmt
> +--echo #
> +
> +CREATE TABLE a (
> +  a_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +  b_id INT(20) UNSIGNED NULL DEFAULT NULL,
> +  c_id VARCHAR(255) NULL DEFAULT NULL,
> +  PRIMARY KEY (a_id)
> +)
> +COLLATE = 'utf8_general_ci'
> +ENGINE = InnoDB;
> +
> +CREATE TABLE b (
> +b_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +c_id VARCHAR(255) NULL DEFAULT NULL,
> +PRIMARY KEY (b_id),
> +INDEX idx_c_id (c_id)
> +)
> +COLLATE = 'utf8_general_ci'
> +ENGINE = InnoDB;
> +
> +INSERT INTO a (b_id, c_id) VALUES (1, NULL);
> +INSERT INTO b (c_id) VALUES (NULL);
> +INSERT INTO b (c_id) VALUES (NULL);
> +
> +select b_id from a;
> +UPDATE a SET b_id = (SELECT b.b_id FROM b b WHERE b.c_id = a.c_id);
> +select b_id from a;
> +drop table a,b;
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 85bfb6b..3543cc2 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -9363,34 +9363,54 @@ static void add_not_null_conds(JOIN *join)
>  not_null_item is the t1.f1, but it's referred_tab is 0.
>*/
>if (!referred_tab)
> -continue;
> -  if (!(notnull= new (join->thd->mem_root)
> +  {
> +if (!(notnull= new (join->thd->mem_root)
>  Item_func_isnotnull(join->thd, item)))
> -DBUG_VOID_RETURN;
> -  /*
> -We need to do full fix_fields() call here in order to have 
> correct
> -notnull->const_item(). This is needed e.g. by test_quick_select 
> -when it is called from make_join_select after this function is 
> -called.
> -  */
Note this as (LOC1)
> -  if (notnull->fix_fields(join->thd, ¬null))
> -DBUG_VOID_RETURN;
> -  DBUG_EXECUTE("where",print_where(notnull,
> -   
> referred_tab->table->alias.c_ptr(),
> -   QT_ORDINARY););
> -  if (!tab->first_inner)
> -   {
> -COND *new_cond= referred_tab->join == join ? 
> -  

Re: [Maria-developers] [Commits] 787610a: MDEV-10766: Queries which start with WITH clause do not get inserted into query cache

[Sergey Petrunia]

Hi Varun,

On Fri, Feb 03, 2017 at 08:20:51PM +0530, Varun wrote:
> revision-id: 787610a1f5cec04311fd16f6c1de87171e5cfdd4 
> (mariadb-10.2.3-159-g787610a)
> parent(s): bc12d993d7bc94a9533028a258afc7e4ceb21e92
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-02-03 20:17:16 +0530
> message:
> 
> MDEV-10766: Queries which start with WITH clause do not get inserted into 
> query cache
> 
> Added conditions so that the WITH queries are also added to the query cache
> 
> ---
>  mysql-test/r/query_cache.result | 30 ++
>  mysql-test/t/query_cache.test   | 14 ++
>  sql/sql_cache.cc|  5 -
>  3 files changed, 48 insertions(+), 1 deletion(-)
> 
> diff --git a/mysql-test/r/query_cache.result b/mysql-test/r/query_cache.result
> index d34c96c..fa169d4 100644
> --- a/mysql-test/r/query_cache.result
> +++ b/mysql-test/r/query_cache.result
> @@ -2136,6 +2136,36 @@ Qcache_hits1
>  use test;
>  drop database `foo.bar`;
>  End of 10.0 tests
> +FLUSH USER_STATISTICS;
> +FLUSH CLIENT_STATISTICS;

I'm wondering what is the reason for the above? Qcache_XXX status variables 
are flushed with FLUSH STATUS.

> +flush status;
> +show status like "Qcache_inserts";
> +Variable_nameValue
> +Qcache_inserts   0
> +create table t1 (i int);
> +with cte as (select * from t1) select * from cte;
> +i
> +show status like "Qcache_queries_in_cache";
> +Variable_nameValue
> +Qcache_queries_in_cache  1
> +show status like "Qcache_inserts";
> +Variable_nameValue
> +Qcache_inserts   1
> +show status like "Qcache_hits";
> +Variable_nameValue
> +Qcache_hits  0
> +with cte as (select * from t1) select * from cte;
> +i
> +show status like "Qcache_queries_in_cache";
> +Variable_nameValue
> +Qcache_queries_in_cache  1
> +show status like "Qcache_inserts";
> +Variable_nameValue
> +Qcache_inserts   1
> +show status like "Qcache_hits";
> +Variable_nameValue
> +Qcache_hits  1
> +drop table t1;
>  restore defaults
>  SET GLOBAL query_cache_type= default;
>  SET GLOBAL query_cache_size= default;

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Archive Storage Engine

[Sergey Petrunia]

Hi Varun,

On Tue, Feb 07, 2017 at 10:19:50PM +0100, Sergei Golubchik wrote:
> 
> I almost replied to your email with "it's impossible, Archive did not
> have HA_RECORD_MUST_BE_CLEAN_ON_WRITE in the table_flags(), no engine
> did. So removal could not have changed anything".
> 
> But then I noticed that the code - by some old typo - had "&&" not "&".
> So it was always doing restore_record() in valgrind-aware binaries.
> For all engines. Apparently, it was never the original intention.
> 
> Note that in release binaries HAVE_valgrind is not defined, and the
> record is not bzero-ed even in 10.1 and earlier versions. This is a bug
> in Archive engine, in all versions, not in 10.2 only. In should not
> store the uninitialized tail of VARCHAR columns.
> 

Note that all this only happens in the "old" data format of the archive
tables. The new data format stores VARCHAR columns as variable-length data
and doesnt have this problem.

As for the old format, we can't change it. 
I guess the solution is that ha_archive::pack_row_v1() should fill the padding 
bytes in record_buffer->buffer (not in 'uchar *record' argument) with zeros.

I've did varchar packing for other storage engines, so I'll provide here some
pointers on how to do it:

* The data in "uchar *record" argument of pack_row_v1() is in the data format
  that we call "table->record[0] format"

* In that data format, VARCHAR column has a fixed space.
It starts with this offset:
(field->ptr - table->record[0])
and takes field->pack_length_in_rec() bytes.
  
* field->is_real_null(record - table->record[0]) tells whether the field has a
NULL value. for NULLs, all varchar value is garbage

* for non-NULL values:
field_varchar->length_bytes  shows how many bytes store the value length.
Then, this many bytes have the actual value. The rest is garbage.

* You only need to handle VARCHARs, field->real_type() == MYSQL_TYPE_VARCHAR

Hope that helps.

> On Feb 07, Varun Gupta wrote:
> > Hi,
> > 
> > The commit is:
> > 
> > commit c697ddc315429c56dec567b2fe85cfe5c03834ea
> > Author: Sergei Golubchik 
> > Date:   Mon Dec 12 15:47:51 2016 +0100
> > 
> > cleanup: remove unused handler table flag
> > 
> > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc
> > 
> > index 901fd48..9c21cb7 100644
> > --- a/sql/sql_insert.cc
> > +++ b/sql/sql_insert.cc
> > 
> > @@ -955,12 +955,7 @@ bool mysql_insert(THD *thd,TABLE_LIST *table_list,
> >  be overwritten by fill_record() anyway (and fill_record() does
> > not
> >  use default values in this case).
> >*/
> > -#ifdef HAVE_valgrind
> > -  if (table->file->ha_table_flags() && 
> > HA_RECORD_MUST_BE_CLEAN_ON_WRITE)
> > -restore_record(table,s->default_values);   // Get empty record
> > -  else
> > -#endif
> > -table->record[0][0]= share->default_values[0];
> > +  table->record[0][0]= share->default_values[0];
> > 
> > On Sun, Feb 5, 2017 at 8:10 PM, Sergei Golubchik  wrote:
> > 
> > > Hi, Varun!
> > >
> > > On Feb 05, Varun Gupta wrote:
> > > > Hi,
> > > >
> > > > Here is the issue https://jira.mariadb.org/browse/MDEV-11645 on which I
> > > > need suggestions
> > > >
> > > > The observations made are:
> > > >
> > > > - in 10.1, varchar endspace was bzero-ed
> > > > archive depended on it to not pack garbage
> > > >
> > > > - in 10.2, varchar endspace is not bzero-ed
> > > >  which breaks the archive SE.
> > > >
> > > > Was this change intended?
> > > >
> > > > If yes, then we should update ha_archive to fill the garbage parts
> > > > of the buffer with zeroes that will cause it to be (slightly?)
> > > > slower  but there's no other way.
> > >
> > > Could you find what commit caused it? git bisect is quite useful for
> > > that.
> > >

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 15fde78: MDEV-10731: Wrong NULL match results in "Subquery returns more than 1 row" (error code 1242)

[Sergey Petrunia]

Hi Varun,

On Thu, Feb 09, 2017 at 10:37:17PM +0530, Varun wrote:
> revision-id: 15fde78ce812c8db91340936941b986fb28ce1ad 
> (mariadb-10.1.20-82-g15fde78)
> parent(s): 99b2de92c6214ddd73eba35c935f490eedf87a26
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-02-09 22:36:03 +0530
> message:
> 
> MDEV-10731: Wrong NULL match results in "Subquery returns more than 1 row" 
> (error code 1242)
> 
> Post review fixes

As far as I understand it's not Post-review fixes, it's a complete patch.
Please use an appropriate comment.

> 
> ---
>  mysql-test/r/update_innodb.result | 26 ++
>  mysql-test/t/update_innodb.test   | 26 ++
>  sql/sql_select.cc | 19 +++
>  3 files changed, 63 insertions(+), 8 deletions(-)
> 
> diff --git a/mysql-test/r/update_innodb.result 
> b/mysql-test/r/update_innodb.result
> index 88c86c5..9ba6699 100644
> --- a/mysql-test/r/update_innodb.result
> +++ b/mysql-test/r/update_innodb.result
> @@ -29,3 +29,29 @@ CREATE ALGORITHM=UNDEFINED VIEW `v1` AS select `t4`.`c1` 
> AS `c1`,`t4`.`c2` AS `c
>  UPDATE t1 a JOIN t2 b ON a.c1 = b.c1 JOIN v1 vw ON b.c2 = vw.c1 JOIN t3 del 
> ON vw.c2 = del.c2 SET a.c2 = ( SELECT max(t.c1) FROM t3 t, v1 i WHERE del.c2 
> = t.c2 AND vw.c3 = i.c3 AND t.c3 = 4 ) WHERE a.c2 IS NULL OR a.c2 < 
> '2011-05-01';
>  drop view v1;
>  drop table t1,t2,t3,t4;
> +#
> +# MDEV-10232 Scalar result of subquery changes after adding an outer select 
> stmt
> +#
> +CREATE TABLE t1 (
> +a_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +b_id INT(20) UNSIGNED NULL DEFAULT NULL,
> +c_id VARCHAR(255) NULL DEFAULT NULL,
> +PRIMARY KEY (a_id))COLLATE = 'utf8_general_ci' ENGINE = InnoDB;
> +CREATE TABLE t2 (
> +b_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +c_id VARCHAR(255) NULL DEFAULT NULL,
> +PRIMARY KEY (b_id),
> +INDEX idx_c_id (c_id))COLLATE = 'utf8_general_ci' ENGINE = InnoDB;
> +INSERT INTO t1 (b_id, c_id) VALUES (NULL, NULL);
> +INSERT INTO t2 (c_id) VALUES (NULL);
> +INSERT INTO t2 (c_id) VALUES (NULL);
> +SELECT * FROM T1;
> +a_id b_idc_id
> +1NULLNULL
> +SELECT t2.b_id FROM t1,t2 WHERE t2.c_id = t1.c_id;
> +b_id
> +UPDATE t1 SET b_id = (SELECT t2.b_id FROM t2 t2 WHERE t2.c_id = t1.c_id);
> +SELECT * FROM T1;
> +a_id b_idc_id
> +1NULLNULL
> +drop table t1,t2;
> diff --git a/mysql-test/t/update_innodb.test b/mysql-test/t/update_innodb.test
> index 67c356c..d4fb8b7 100644
> --- a/mysql-test/t/update_innodb.test
> +++ b/mysql-test/t/update_innodb.test
> @@ -37,3 +37,29 @@ UPDATE t1 a JOIN t2 b ON a.c1 = b.c1 JOIN v1 vw ON b.c2 = 
> vw.c1 JOIN t3 del ON v
>  
>  drop view v1;
>  drop table t1,t2,t3,t4;
> +
> +--echo #
> +--echo # MDEV-10232 Scalar result of subquery changes after adding an outer 
> select stmt
> +--echo #
> +
> +CREATE TABLE t1 (
> +  a_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +  b_id INT(20) UNSIGNED NULL DEFAULT NULL,
> +  c_id VARCHAR(255) NULL DEFAULT NULL,
> +  PRIMARY KEY (a_id))COLLATE = 'utf8_general_ci' ENGINE = InnoDB;
> +
> +CREATE TABLE t2 (
> +  b_id INT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
> +  c_id VARCHAR(255) NULL DEFAULT NULL,
> +  PRIMARY KEY (b_id),
> +  INDEX idx_c_id (c_id))COLLATE = 'utf8_general_ci' ENGINE = InnoDB;
> +
> +INSERT INTO t1 (b_id, c_id) VALUES (NULL, NULL);
> +INSERT INTO t2 (c_id) VALUES (NULL);
> +INSERT INTO t2 (c_id) VALUES (NULL);
> +
> +SELECT * FROM T1;
> +SELECT t2.b_id FROM t1,t2 WHERE t2.c_id = t1.c_id;
> +UPDATE t1 SET b_id = (SELECT t2.b_id FROM t2 t2 WHERE t2.c_id = t1.c_id);
> +SELECT * FROM T1;
> +drop table t1,t2;
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index e37c1b8..bc9a80f 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -9367,8 +9367,6 @@ static void add_not_null_conds(JOIN *join)
>  UPDATE t1 SET t1.f2=(SELECT MAX(t2.f4) FROM t2 WHERE 
> t2.f3=t1.f1);
>  not_null_item is the t1.f1, but it's referred_tab is 0.
>*/
> -  if (!referred_tab)
> -continue;
>if (!(notnull= new (join->thd->mem_root)
>  Item_func_isnotnull(join->thd, item)))
>  DBUG_VOID_RETURN;
> @@ -9380,16 +9378,21 @@ static void add_not_null_conds(JOIN *join)
>*/
>if (notnull->fix_fields(join->thd, ¬null))
>  DBUG_VOID_RETURN;
> -  DBUG_EXECUTE("where",print_where(notnull,
> -   
> referred_tab->table->alias.c_ptr(),
> -   QT_ORDINARY););
> +
> +  if(referred_tab)
> +DBUG_EXECUTE("where",print_where(notnull,
> + 
> referred_tab->table->alias.c_ptr(),
> + QT_ORDINARY););
> +  else
> +DBUG_EXECUTE("where",print_where(notnull,
> + "outer_ref_cond",QT_ORDINARY););

Please take the re-factoring further and have o

Re: [Maria-developers] [Commits] 13ebc9c: MDEV-11645: archive.archive fails in buildbot with valgrind (Use of uninitialised value)

[Sergey Petrunia]

Hi Varun,

On Mon, Feb 13, 2017 at 03:47:57AM +0530, Varun wrote:
> revision-id: 13ebc9c826de080e416071b7881585b45922bef9 
> (mariadb-10.2.3-185-g13ebc9c)
> parent(s): 3ae038b732ce503fb839e9095355e05f5c6866f9
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-02-13 03:45:59 +0530
> message:
> 
> MDEV-11645: archive.archive fails in buildbot with valgrind (Use of 
> uninitialised value)
> 
> Set null values to record_buffer->buffer for the VARCHAR fields
> 
> ---
>  storage/archive/ha_archive.cc | 21 +
>  1 file changed, 21 insertions(+)
> 
> diff --git a/storage/archive/ha_archive.cc b/storage/archive/ha_archive.cc
> index 9d6d100..329e3d5 100644
> --- a/storage/archive/ha_archive.cc
> +++ b/storage/archive/ha_archive.cc
> @@ -372,10 +372,31 @@ int Archive_share::write_v1_metafile()
>  
>  unsigned int ha_archive::pack_row_v1(uchar *record)
>  {
> +  uint offset= 0,length;
>uint *blob, *end;
>uchar *pos;
>DBUG_ENTER("pack_row_v1");
>memcpy(record_buffer->buffer, record, table->s->reclength);
> +  for(Field** field= table->field; (*field) ; field++)
> +  {
> +
> +Field *fld= *field;
> +
> +if(fld->type() == MYSQL_TYPE_VARCHAR)
> +{
> +  uint start= (fld->ptr - table->record[0]);
> +  uint index_end= start+fld->field_length;
There is no reason to compute the above variables when the field value is an
SQL NULL.
Please move them into the if.
Please also use ptrdiff_t datatype instead of uint.

A different question: did you check what happens when one attempts to write an
SQL NULL value?

> +
> +  if(!(fld->is_real_null(record - table->record[0])))
> +  {
> +Field_varstring *const field_var = (Field_varstring *)fld;
> +offset = field_var->data_length() + field_var->length_size();
Coding style says there should be no space before the assignment. Please fix
the above two lines.

> +
> +for(uint index= start + offset; index <= index_end ; index++)
> +record_buffer->buffer[index]=0;

Is there any reason you're not using bzero or memset call here instead? (please
check with Serg which one is guaranteed to be available)

> +  }
> +}
> +  }
>pos= record_buffer->buffer + table->s->reclength;
>for (blob= table->s->blob_field, end= blob + table->s->blob_fields;
> blob != end; blob++)

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] e0a1e63: MDEV-9619: Assertion `null_ref_table' failed in virtual table_map Item_direct_view_ref::used_tables() const on 2nd execution of PS

[Sergey Petrunia]

Hi Sanja,

Two questions:
1. Why does the patch define Item_ptr type twice?

2. (The big one) In class Item_in_subselect, I see your comment made in 2015:

...
public:
  Item *left_expr;
  /*
Important for PS/SP: left_expr_orig is the item that left_expr originally
pointed at. That item is allocated on the statement arena, while
left_expr could later be changed to something on the execution arena.
  */
  Item *left_expr_orig;


Considering this comment, is it really safe to keep pointers to left_expr?
Should left_expr_orig be used instead?

On Sat, Feb 18, 2017 at 05:47:32PM +0100, Oleksandr Byelkin wrote:
> revision-id: e0a1e632e4e6084cd989f217943618f4e9ac9179 
> (mariadb-5.5.54-25-ge0a1e63)
> parent(s): 29d78dbb44ee9890b6bc28873344f20fc9157928
> committer: Oleksandr Byelkin
> timestamp: 2017-02-18 17:47:31 +0100
> message:
> 
> MDEV-9619: Assertion `null_ref_table' failed in virtual table_map 
> Item_direct_view_ref::used_tables() const on 2nd execution of PS
> 
> Refer left expression indirectly in case it changes from execution to 
> execution.
> 
> ---
>  mysql-test/r/ps.result | 24 
>  mysql-test/t/ps.test   | 22 ++
>  sql/item.h |  2 ++
>  sql/opt_subselect.cc   | 10 +-
>  sql/table.h|  3 ++-
>  5 files changed, 55 insertions(+), 6 deletions(-)
> 
> diff --git a/mysql-test/r/ps.result b/mysql-test/r/ps.result
> index f954583..7233fe8 100644
> --- a/mysql-test/r/ps.result
> +++ b/mysql-test/r/ps.result
> @@ -4173,4 +4173,28 @@ Warnings:
>  Note 1003select `test`.`t1`.`id` AS `id`,`test`.`t1`.`c` AS `c` from 
> `test`.`t1` where 0
>  deallocate prepare stmt2;
>  drop table t1;
> +#
> +# MDEV-9619: Assertion `null_ref_table' failed in virtual
> +# table_map Item_direct_view_ref::used_tables() const on 2nd
> +# execution of PS
> +#
> +CREATE TABLE t1 (f1 VARCHAR(10)) ENGINE=MyISAM;
> +CREATE ALGORITHM=MERGE VIEW v1 AS SELECT * FROM t1;
> +INSERT INTO t1 VALUES ('a'),('b');
> +CREATE TABLE t2 (f2 VARCHAR(10)) ENGINE=MyISAM;
> +INSERT INTO t2 VALUES ('c'),('d');
> +PREPARE stmt FROM "SELECT * FROM v1 WHERE f1 = SOME ( SELECT f2 FROM t2 )";
> +EXECUTE stmt;
> +f1
> +EXECUTE stmt;
> +f1
> +insert into t1 values ('c');
> +EXECUTE stmt;
> +f1
> +c
> +EXECUTE stmt;
> +f1
> +c
> +drop view v1;
> +drop table t1,t2;
>  # End of 5.5 tests
> diff --git a/mysql-test/t/ps.test b/mysql-test/t/ps.test
> index cfd810f..627a9bf 100644
> --- a/mysql-test/t/ps.test
> +++ b/mysql-test/t/ps.test
> @@ -3714,4 +3714,26 @@ deallocate prepare stmt2;
>  
>  drop table t1;
>  
> +--echo #
> +--echo # MDEV-9619: Assertion `null_ref_table' failed in virtual
> +--echo # table_map Item_direct_view_ref::used_tables() const on 2nd
> +--echo # execution of PS
> +--echo #
> +
> +CREATE TABLE t1 (f1 VARCHAR(10)) ENGINE=MyISAM;
> +CREATE ALGORITHM=MERGE VIEW v1 AS SELECT * FROM t1;
> +INSERT INTO t1 VALUES ('a'),('b');
> +
> +CREATE TABLE t2 (f2 VARCHAR(10)) ENGINE=MyISAM;
> +INSERT INTO t2 VALUES ('c'),('d');
> +
> +PREPARE stmt FROM "SELECT * FROM v1 WHERE f1 = SOME ( SELECT f2 FROM t2 )";
> +EXECUTE stmt;
> +EXECUTE stmt;
> +insert into t1 values ('c');
> +EXECUTE stmt;
> +EXECUTE stmt;
> +
> +drop view v1;
> +drop table t1,t2;
>  --echo # End of 5.5 tests
> diff --git a/sql/item.h b/sql/item.h
> index 89155ac..dff7bc4 100644
> --- a/sql/item.h
> +++ b/sql/item.h
> @@ -1543,6 +1543,8 @@ class Item {
>virtual void mark_as_condition_AND_part(TABLE_LIST *embedding) {};
>  };
>  
> +typedef Item (*Item_ptr);
> +
>  
>  /**
>Compare two Items for List::add_unique()
> diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc
> index 5137d8a..7f26d01 100644
> --- a/sql/opt_subselect.cc
> +++ b/sql/opt_subselect.cc
> @@ -1602,7 +1602,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, 
> Item_in_subselect *subq_pred)
>  
>if (subq_pred->left_expr->cols() == 1)
>{
> -nested_join->sj_outer_expr_list.push_back(subq_pred->left_expr);
> +nested_join->sj_outer_expr_list.push_back(&subq_pred->left_expr);
>  /*
>Create Item_func_eq. Note that
>1. this is done on the statement, not execution, arena
> @@ -1624,7 +1624,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, 
> Item_in_subselect *subq_pred)
>  for (uint i= 0; i < subq_pred->left_expr->cols(); i++)
>  {
>nested_join->sj_outer_expr_list.push_back(subq_pred->left_expr->
> -element_index(i));
> +addr(i));
>Item_func_eq *item_eq= 
>  new Item_func_eq(subq_pred->left_expr->element_index(i), 
>   subq_lex->ref_pointer_array[i]);
> @@ -3181,8 +3181,8 @@ void restore_prev_sj_state(const table_map 
> remaining_tables,
>  ulonglong get_bound_sj_equalities(TABLE_LIST *sj_nest, 
>table_map remaining_tables)
>  {
> -  List_iterator li(sj_nest->nested_join->sj_outer_expr_list);
> -  Ite

[Maria-developers] On constructing ref access from mis-matched charset comparisons

[Sergey Petrunia]

Hi Alexander,

I've got a question about mis-matched charset comparisons and ref access method.

== Short form ==
I know that VARCHAR comparisons over mis-matching charsets cannot be used for 
constructing index lookups into the index over the "narrower" character set
column. 
But is this a real limitation or just an optimizer deficiency?

If it is the latter, do people hit it in the real world sufficiently often to
warrant lifting it?

== Long form ==

=== Example dataset ===

create table t0 (
  a varchar(32) character set utf8 collate utf8_general_ci
);

create table t1 (
  a varchar(32) character set latin1 collate latin1_swedish_ci,
  col1 varchar(32),
  key(a)
);

create table t2 (
  a varchar(32) character set utf8 collate utf8_general_ci,
  col1 varchar(32),
  key(a)
);

insert into t0 select a from ten;
insert into t1 select a,a from one_k;
insert into t2 select a,a from one_k;

=== Queries ===

MariaDB [test]> explain 
-> select * from t0, t2 where t0.a=t2.a;
+--+-+---+--+---+--+-+---+--+-+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  
 | rows | Extra   |
+--+-+---+--+---+--+-+---+--+-+
|1 | SIMPLE  | t0| ALL  | NULL  | NULL | NULL| NULL 
 |   10 | Using where |
|1 | SIMPLE  | t2| ref  | a | a| 99  | 
test.t0.a |1 | |
+--+-+---+--+---+--+-+---+--+-+

Ok, good.

MariaDB [test]> explain 
-> select * from t0, t1 where t0.a=t1.a;
+--+-+---+--+---+--+-+--+--+-+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | 
rows | Extra   |
+--+-+---+--+---+--+-+--+--+-+
|1 | SIMPLE  | t0| ALL  | NULL  | NULL | NULL| NULL |   
10 | |
|1 | SIMPLE  | t1| ALL  | NULL  | NULL | NULL| NULL | 
1000 | Using where; Using join buffer (flat, BNL join) |
+--+-+---+--+---+--+-+--+--+-+
2 rows in set (0.00 sec)

Oops, the index t1(a) is not used. FORMAT=JSON shows the reason:

  "attached_condition": "(t0.a = convert(t1.a using utf8))"

Still it feels like the optimizer could

* Try to convert the value of t0.a into latin1.  
* If it can't be represented in latin1, then we know that no row
  in t1 has t1.a=t0.a.
* make a lookup on t1.a= convert(t0.a using latin1) , using t1.a's collation
  rules.

Is this true [for some charsets] ?

If yes, is it worth doing?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] c6d172f: MDEV-11078: NULL NOT IN (non-empty subquery) should never return results

[Sergey Petrunia]

Hi Varun,

Things that look apparently wrong:

1. The patch checks left_expr->null_value without first having made a call 
that will fill that member.
The calling convention is something like this:

  item->val_XXX()  // val_str or val_int()
  // check item->null_value.

If one doesn't need string or integer value, there is item->is_null() call.

2. The patch checks null_value for any kind of left_expr. This is wrong. For 
example, consider a query:

  select
t1.col IN (select t2.col FROM t2 )
  from t1

Here, it doesn't make any sense to check {t1.col}->null_value at query
planning/rewrite time.  The value (and NULL-ness) of t1.col is different for 
different rows of t1.

As discussed on Slack: please only check for NULL value for items which are 
- constant,
- and "not expensive"
and so have a fixed value which we are allowed to compute at query optimization
phase.

3. I see the patch handles single-value comparisons like

expr IN (select inner_expr FROM ...)

but what about tuple-based comparisons like:

(expr1, expr2) IN (select inner_expr1, inner_expr2 FROM ...)

?

I guess they should have a similar issue. If they do not, we need
- an explanation why
- test coverage

On Sun, Feb 26, 2017 at 09:04:40PM +0530, Varun wrote:
> revision-id: c6d172f785f31641832087a432d1966b67efc47a 
> (mariadb-10.2.3-283-gc6d172f)
> parent(s): 78153cf641aea41166d3e79ae99b57b154f6a027
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-02-26 21:03:29 +0530
> message:
> 
> MDEV-11078: NULL NOT IN (non-empty subquery) should never return results
> 
> Setting the value of cond guard variables during the creation of Tricond Item 
> for null values
> 
> ---
>  mysql-test/r/subselect4.result | 31 +++
>  mysql-test/t/subselect4.test   | 18 ++
>  sql/item_subselect.cc  | 10 ++
>  3 files changed, 59 insertions(+)
> 
> diff --git a/mysql-test/r/subselect4.result b/mysql-test/r/subselect4.result
> index 6bbb80c..c3bd50a 100644
> --- a/mysql-test/r/subselect4.result
> +++ b/mysql-test/r/subselect4.result
> @@ -2454,3 +2454,34 @@ x
>  c1
>  1
>  drop table t1;
> +#
> +# MDEV-11078: NULL NOT IN (non-empty subquery) should never return results
> +#
> +create table t1(a int,b int);
> +create table t2(a int,b int);
> +insert into t1 value (NULL,2);
> +select 1 in (1,2, NULL);
> +1 in (1,2, NULL)
> +1
> +select (null)  in (select 1 from t1);
> +(null)  in (select 1 from t1)
> +NULL
> +select (null)  in (select 1 from t2);
> +(null)  in (select 1 from t2)
> +0
> +select 1 in (select 1 from t1);
> +1 in (select 1 from t1)
> +1
> +select 1 in (select 1 from t2);
> +1 in (select 1 from t2)
> +0
> +select 1 from dual where null in (select 1 from t1);
> +1
> +select 1 from dual where null in (select 1 from t2);
> +1
> +select 1 from dual where null not in (select 1 from t1);
> +1
> +select 1 from dual where null not in (select 1 from t2);
> +1
> +1
> +drop table t1,t2;
> diff --git a/mysql-test/t/subselect4.test b/mysql-test/t/subselect4.test
> index 253160c..785b79d 100644
> --- a/mysql-test/t/subselect4.test
> +++ b/mysql-test/t/subselect4.test
> @@ -2009,3 +2009,21 @@ insert into t1 values(2,1),(1,2);
>  select (select c1 from t1 group by c1,c2 order by c1 limit 1) as x;
>  (select c1 from t1 group by c1,c2 order by c1 limit 1);
>  drop table t1;
> +
> +--echo #
> +--echo # MDEV-11078: NULL NOT IN (non-empty subquery) should never return 
> results
> +--echo #
> +
> +create table t1(a int,b int);
> +create table t2(a int,b int);
> +insert into t1 value (NULL,2);
> +select 1 in (1,2, NULL);
> +select (null)  in (select 1 from t1);
> +select (null)  in (select 1 from t2);
> +select 1 in (select 1 from t1);
> +select 1 in (select 1 from t2);
> +select 1 from dual where null in (select 1 from t1);
> +select 1 from dual where null in (select 1 from t2);
> +select 1 from dual where null not in (select 1 from t1);
> +select 1 from dual where null not in (select 1 from t2);
> +drop table t1,t2;
> diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc
> index 94bc71c..b9c798e 100644
> --- a/sql/item_subselect.cc
> +++ b/sql/item_subselect.cc
> @@ -2113,6 +2113,8 @@ Item_in_subselect::create_single_in_to_exists_cond(JOIN 
> *join,
>  We can encounter "NULL IN (SELECT ...)". Wrap the added condition
>  within a trig_cond.
>*/
> +  if(left_expr->null_value)
> +set_cond_guard_var(0,FALSE);
>item= new (thd->mem_root) Item_func_trig_cond(thd, item, 
> get_cond_guard(0));
>  }
>  
> @@ -2137,6 +2139,9 @@ Item_in_subselect::create_single_in_to_exists_cond(JOIN 
> *join,
>   having= new (thd->mem_root) Item_is_not_null_test(thd, this, having);
>  if (left_expr->maybe_null)
>  {
> +  if(left_expr->null_value)
> +set_cond_guard_var(0,FALSE);
> +
>if (!(having= new (thd->mem_root) Item_func_trig_cond(thd, having,
>  
> 

Re: [Maria-developers] [Commits] 00c30e5: MDEV-11645: archive.archive fails in buildbot with valgrind (Use of uninitialised value)

[Sergey Petrunia]

Hi Varun,


Please
* fix the commit comment
* add a comment
* fix the coding style 

as indicated below.

(and also please conform to the coding style in the future. If its requirements
are not clear, lets sit down once and discuss them)

On Wed, Mar 01, 2017 at 07:50:01AM +0530, Varun wrote:
> revision-id: 00c30e5a234fbd01484cd069c01e97c075025323 
> (mariadb-10.2.3-283-g00c30e5)
> parent(s): b13cee83f9cfcffa98d227c492411a9acec85f42
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-03-01 07:46:26 +0530
> message:
> 
> MDEV-11645: archive.archive fails in buildbot with valgrind (Use of 
> uninitialised value)
> 
> Set null values to record_buffer->buffer for the VARCHAR fields

What "NULL values"?? The fix is filling with zeros the space beyond the end of
VARCHAR values. VARCHAR NULLs are actually ok already as NULL value has its 
buffer 
filled with zeros.

Please come up with a comment that describes the ix.
> 
> ---
>  storage/archive/ha_archive.cc | 15 +++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/storage/archive/ha_archive.cc b/storage/archive/ha_archive.cc
> index 9d6d100..5599b43 100644
> --- a/storage/archive/ha_archive.cc
> +++ b/storage/archive/ha_archive.cc
> @@ -372,10 +372,25 @@ int Archive_share::write_v1_metafile()
>  
>  unsigned int ha_archive::pack_row_v1(uchar *record)
>  {
> +  uint offset;
>uint *blob, *end;
>uchar *pos;
>DBUG_ENTER("pack_row_v1");
>memcpy(record_buffer->buffer, record, table->s->reclength);
Please add here a comment about what this loop is doing.

> +  for(Field** field= table->field; (*field) ; field++)
space needed in 'for ('

> +  {
> +Field *fld= *field;
> +if (fld->type() == MYSQL_TYPE_VARCHAR)
> +{
> +  if (!(fld->is_real_null(record - table->record[0])))
> +  {
> +ptrdiff_t  start= (fld->ptr - table->record[0]);
> +Field_varstring *const field_var= (Field_varstring *)fld;
> +offset= field_var->data_length() + field_var->length_size();
> +bzero(record_buffer->buffer + start + offset, fld->field_length - 
> offset + 1);
The above line is longer than 80 chars. should be broken into:

bzero(record_buffer->buffer + start + offset, 
  fld->field_length - offset + 1);


> +  }
> +}
> +  }
>pos= record_buffer->buffer + table->s->reclength;
>for (blob= table->s->blob_field, end= blob + table->s->blob_fields;
> blob != end; blob++)

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] MDEV-10306 Wrong results with combination of CONCAT, SUBSTR and CONVERT in subquery

[Sergey Petrunia]

Hello,

On Wed, Mar 01, 2017 at 01:56:24PM +0100, Sergei Golubchik wrote:
> Hi, Alexander!
> 
> On Feb 28, Alexander Barkov wrote:
> 
> > commit af8887b86ccbaea8782cf54fe445cf53aaef7c06
> > Author: Alexander Barkov 
> > Date:   Tue Feb 28 10:28:09 2017 +0400
> > 
> > MDEV-10306 Wrong results with combination of CONCAT, SUBSTR and CONVERT 
> > in subquery
> > 
> > The bug happens because of a combination of unfortunate circumstances:
> > 
> > 1. Arguments args[0] and args[2] of Item_func_concat point recursively
> > (through Item_direct_view_ref's) to the same Item_func_conv_charset.
> > Both args[0]->args[0]->ref[0] and args[2]->args[0]->ref[0] refer to
> > this Item_func_conv_charset.
> > 
> > 2. When Item_func_concat::args[0]->val_str() is called,
> > Item_func_conv_charset::val_str() writes its result to
> > Item_func_conc_charset::tmp_value.
> > 
> > 3. Then, for optimization purposes (to avoid copying),
> > Item_func_substr::val_str() initializes Item_func_substr::tmp_value
> > to point to the buffer fragment owned by 
> > Item_func_conv_charset::tmp_value
> > Item_func_substr::tmp_value is returned as a result of
> > Item_func_concat::args[0]->val_str().
> > 
> > 4. Due to optimization to avoid memory reallocs,
> > Item_func_concat::val_str() remembers the result of args[0]->val_str()
> > in "res" and further uses "res" to collect the return value.
> > 
> > 5. When Item_func_concat::args[2]->val_str() is called,
> > Item_func_conv_charset::tmp_value gets overwritten (see #1),
> > which effectively overwrites args[0]'s Item_func_substr::tmp_value (see 
> > #3),
> > which effectively overwrites "res" (see #4).
> > 
> > The fix marks Item_func_substr::tmp_value as a constant string, which
> > tells Item_func_concat::val_str "Don't use me as the return value, you 
> > cannot
> > append to me because I'm pointing to a buffer owned by some other 
> > String".
> 
> This pretty much looks like a hack, that makes the bug disappear in this
> particular test case.
> 
> What if SUBSTR() wasn't used? CONCAT would still modify
> args[0]->tmp_value, and it would be overwritten by args[2]->val_str().
> 
> On the other hand, if you remove args[2] from the test case, then CONCAT
> can safely modify args[0]'s buffer and marking SUBSTR as const would
> prevent a valid optimization.
> 
> So, I see few possible approaches to this and other similar queries:
> 
> 1. We specify that no Item's val method can modify the buffer of the
>arguments. That is, CONCAT will always have to copy. SUBSTR won't
>need to copy, because it doesn't modify the buffer, it only returns a
>pointer into it.
> 
> 2. May be #1 is not strict enough, and we'll need to disallow pointers
>into the arguments' buffer too. Because, perhaps, args[2]->val_str()
>could realloc and then the pointer will become invalid.
> 
> 3. A different approach would be to disallow one item to appear twice in
>an expression. No idea how to do that.
> 
> 4. A variand of #3, an item can appear many times, but it'll be only
>evaluated once per row. That still needs #1, but #2 is unnecessary.
> 
> Opinions? Ideas?

My take on the issue: I read the comment for Item::val_str:

  The caller of this method can modify returned string, but only in case
  when it was allocated on heap, (is_alloced() is true).  This allows
(*)
  the caller to efficiently use a buffer allocated by a child without
  having to allocate a buffer of it's own. The buffer, given to
  val_str() as argument, belongs to the caller and is later used by the
  caller at it's own choosing.
  A few implications from the above:
  - unless you return a string object which only points to your buffer
but doesn't manages it you should be ready that it will be
modified.
  - even for not allocated strings (is_alloced() == false) the caller
can change charset (see Item_func_{typecast/binary}. XXX: is this
a bug?
  - still you should try to minimize data copying and return internal
object whenever possible.

And I see that Item_func_concat::val_str violates the rule (*) here:

=>if (!is_const && res->alloced_length() >= res->length()+res2->length())
  { // Use old buffer
res->append(*res2);

Here I see that
(gdb) p *res
  $81 = {Ptr = 0x7fffcb4203f0 "123---", str_length = 6, Alloced_length = 8,
  extra_alloc = 0, alloced = false, thread_specific = false, str_charset =
  0x175de80 }

(gdb) p res->is_alloced()
  $83 = false

That is, the buffer is "not alloced", but the code modifies it by calling
append().

So, I can get this bug' testcase to pass with this patch:

diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc
index 4ea3075..2e2cecd 100644
--- a/sql/item_strfunc.cc
+++ b/sql/i

Re: [Maria-developers] [GSOC] Table functions (aka SQL functions returning tables)

[Sergey Petrunia]

Hi Pranjal,

On Sun, Mar 19, 2017 at 09:53:08PM +0530, Pranjal Rai wrote:
> My name is Pranjal and I am a 4th year student at IIIT Hyderabad. I am
> interested in working on the project  MDEV-8100
>  "Table functions (aka SQL
> functions returning tables)".
> I read the description of the task and it is quite clear to me. In the
> comments it is advised to consider creating TABLE datatype for the task. Is
> this task along with creation of TABLE datatype doable in the given
> timeline?

I think it is difficult to tell with certainity at this point.

AFAIU the sub-task of adding a table datatype is a nice to have, but should be
put at the end of the schedule:

- It depends on other tasks and no other tasks depend on it
- There are scenarios where MDEV-8100 is usable even if TABLE datatype is not 
  implemented.

I personally would strongly prefer to see work for "MDEV-8100 without 
TABLE-datatype" finished than unfinished "MDEV-8100 with table datatype".

(adding Bar on cc:)

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 96403c5e124: Replace some unsigned integers with size_t to avoid type mismatch.

[Sergey Petrunia]

Hi Marko,

Please find below the review for patch #1 of the "size_t patches":
my comments are marked with 'psergey:'

On Thu, Mar 30, 2017 at 02:39:50PM +0300, marko.mak...@mariadb.com wrote:
> commit 96403c5e1240342948cb88600ece14c0c8b72dbc
> Author: Marko Mäkelä 
> Date:   Sat Mar 11 22:03:31 2017 +0200
> 
> Replace some unsigned integers with size_t to avoid type mismatch.
> 

> diff --git a/sql/key.cc b/sql/key.cc
> index bb10e90..9e8693f 100644
> --- a/sql/key.cc
> +++ b/sql/key.cc
> @@ -1,4 +1,5 @@
>  /* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights 
> reserved.
> +   Copyright (c) 2017, MariaDB Corporation.
>  
> This program is free software; you can redistribute it and/or modify
> it under the terms of the GNU General Public License as published by
> @@ -392,7 +393,7 @@ void field_unpack(String *to, Field *field, const uchar 
> *rec, uint max_length,
>  tmp.length(charpos);
>  }
>  if (max_length < field->pack_length())
> -  tmp.length(min(tmp.length(),max_length));
> +  tmp.length(min(tmp.length(),static_cast(max_length)));
psergey: This is not necessary as tmp.length() returns a 'size_t', but it's
still ok to have I guess.

>  ErrConvString err(&tmp);
>  to->append(err.ptr());
>}
...
> diff --git a/sql/spatial.cc b/sql/spatial.cc
> index 7c9d8bb..399a968 100644
> --- a/sql/spatial.cc
> +++ b/sql/spatial.cc
> @@ -2278,7 +2278,7 @@ int Gis_multi_line_string::geometry_n(uint32 num, 
> String *result) const
>break;
>  data+= length;
>}
> -  return result->append(data, length, (uint32) 0);
> +  return result->append(data, length, static_cast(0));

psergey: isn't that excessive, can one just write "0" there?

>  }
>  
>  
> @@ -2710,8 +2710,9 @@ int Gis_multi_polygon::geometry_n(uint32 num, String 
> *result) const
>} while (--num);
>if (no_data(data, 0))  // We must check last 
> segment
>  return 1;
> -  return result->append(start_of_polygon, (uint32) (data - start_of_polygon),
> - (uint32) 0);
> +  return result->append(start_of_polygon,
> +static_cast(data - start_of_polygon),
> +static_cast(0));

psergey: isn't that excessive, can one just write "0" there?

>  }
>  
>  
...
> diff --git a/sql/sql_string.h b/sql/sql_string.h
> index 18f5f4c..8248f6d 100644
> --- a/sql/sql_string.h
> +++ b/sql/sql_string.h

> @@ -531,16 +531,19 @@ class String
>{
>  Ptr[str_length++] = c;
>}
> -  void q_append2b(const uint32 n)
> +  void q_append2b(size_t n)

psergey: this just appends the two lower bytes. Is this change really needed?

>{
>  int2store(Ptr + str_length, n);
>  str_length += 2;
>}
> -  void q_append(const uint32 n)
> +  void q_append(size_t n)
>{
psergey: This always appends 4 bytes. It is a little bit scary that this
function will optionally discard the upper 4 bytes of the passed values.

As far as I understand its usage, this is for writing the data that should
always be 4 bytes, regardless of the sizeof(size_t) of the platform we're on.

>  int4store(Ptr + str_length, n);
>  str_length += 4;
>}
> +#if SIZEOF_SIZE_T > 4
> +  void q_append(uint32 n) { q_append(static_cast(n)); }
> +#endif
>void q_append(double d)
>{
>  float8store(Ptr + str_length, d);
...
> @@ -669,11 +672,11 @@ class String
>{
>  return !sortcmp(this, other, cs);
>}
> -  void q_net_store_length(ulonglong length)
> +  void q_net_store_length(size_t length)

psergey:
I dont think this should vary depending on sizeof(size_t). This function is
used to construct data to be sent over network, check sql/protocol.cc,
net_send_ok().
Apparently network data should have the same number of bits always.

>{
>  DBUG_ASSERT(Alloced_length >= (str_length + net_length_size(length)));
>  char *pos= (char *) net_store_length((uchar *)(Ptr + str_length), 
> length);
> -str_length= uint32(pos - Ptr);
> +str_length= static_cast(pos - Ptr);
>}
>void q_net_store_data(const uchar *from, size_t length)
>{


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Errors in building debs

[Sergey Petrunia]

Hi Vicențiu,


I'm looking at the last build in bb-10.2-mariarocks

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-trusty-x86/builds/3558/steps/compile/logs/stdio
and I see this:

+ DH_BUILD_DDEBS=1
+ echo
+ debian/autobake-deb.sh
debian/autobake-deb.sh: line 80: [: i686: integer expression expected
debian/autobake-deb.sh: line 80: [i686: command not found

and then it eventually fails with this error:

cp: cannot stat ‘debian/tmp/etc/mysql/conf.d/rocksdb_se.cnf’: No such file or

I assume this is caused by your recent changes?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 39bce67: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'

[Sergey Petrunia]

Hi Varun,

First reaction:

> diff --git a/sql/sql_table.cc b/sql/sql_table.cc
> index 84e4018..63636f9 100644
> --- a/sql/sql_table.cc
> +++ b/sql/sql_table.cc
> @@ -3948,7 +3948,11 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO 
> *create_info,
>key_part_info->fieldnr= field;
>key_part_info->offset=  (uint16) sql_field->offset;
>key_part_info->key_type=sql_field->pack_flag;
> -  uint key_part_length= sql_field->key_length;
> +  uint key_part_length= MY_MAX(sql_field->key_length,
> +   sql_field->pack_length);

What does the above do? It is not clear why MAX should be used. (Let's denote
the compared variables X and Y. Can X>Y and X +
> +  if (!(sql_field->flags & NOT_NULL_FLAG))
> +  key_part_length++;
>  
>if (column->length)
>{

I am concerned that the above may have an undesired effect:
key_part_kength counts NULL-byte while previously it did not.

Let's try this example:

create table t1 (pk int not null primary key,   col1 int,   key(col1) ) 
engine=innodb;

Run it, and put a breakpoint below on this code:

  /* Check if the key segment is partial, set the key flag accordingly */
  if (key_part_length != sql_field->key_length)
key_info->flags|= HA_KEY_HAS_PART_KEY_SEG;


You will see that for filed 'col1'  HA_KEY_HAS_PART_KEY_SEG flag is now set.
This didn't happen before. Is this really intended?

I wasn't able to find what exactly may get wrong, but if I run this query:

explain select * from t1 where col1=1;  

 
then it hangs.
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] e6dabcd: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'

[Sergey Petrunia]

Hi Varun,

So I am debugging the testcase from the bug report. I reach these lines:

> if(i)
> ext_key_length+= key_part->length;

(Why there is no indentation btw?)

and I see:

(gdb) p key_part->length
 $40 = 3072

3072=1024*3, that is, the NULL-byte and length bytes are NOT counted.
The code then proceeds to check if

  pk_length + ext_key_length  <= MAX_KEY_LENGTH

but that check could produce a wrong result if NULL- and length- bytes are not
counted.

The second issue is that I don't see where ext_key_length is zero'ed.  
I try debugging this query:

CREATE TABLE t2 (
  pk INT,
  f1 VARCHAR(3),
  f2 VARCHAR(1024),
  f2a VARCHAR(1024),
  PRIMARY KEY (pk),
  KEY(f2),
  KEY(f2a) 
) ENGINE=InnoDB CHARSET utf8;

I put a breakpoint at this line:

>if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME)
> && (pk_length + ext_key_length  <= 
> MAX_KEY_LENGTH))

(84 chars! where's the coding style again?)
and I observe:

(gdb) print ext_key_length
  $45 = 6144

This is clearly more than any [possible] key length for the table.

So, this particular patch is not a solution (even if the testcase happens to
pass with it).

On Mon, May 08, 2017 at 04:23:11PM +0530, Varun wrote:
> revision-id: e6dabcd3a7e994feaa1a1dc5d0c1a0d621272f72 
> (mariadb-10.2.2-387-ge6dabcd)
> parent(s): 39bce67b7aa27fc7e5458f36838148f6e0f4c3af
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-05-08 16:21:32 +0530
> message:
> 
> MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 
> 'key_buff'
> was corrupted, server crashes in opt_sum_query
> 
> extended keys feature disabled if the length of extended key is longer than 
> MAX_KEY_LEN
> 
> ---
>  sql/table.cc | 8 ++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/sql/table.cc b/sql/table.cc
> index 398383e..554fc78 100644
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -695,7 +695,7 @@ static bool create_key_infos(const uchar *strpos, const 
> uchar *frm_image_end,
>KEY_PART_INFO *key_part= NULL;
>ulong *rec_per_key= NULL;
>KEY_PART_INFO *first_key_part= NULL;
> -  uint first_key_parts= 0;
> +  uint first_key_parts= 0, pk_length=0, ext_key_length=0;
>  
>if (!keys)
>{  
> @@ -765,6 +765,7 @@ static bool create_key_infos(const uchar *strpos, const 
> uchar *frm_image_end,
>keyinfo->algorithm= first_keyinfo->algorithm;
>if (new_frm_ver >= 3)
>  keyinfo->block_size= first_keyinfo->block_size;
> +  pk_length= keyinfo->key_length;
>  }
>  
>  keyinfo->key_part=key_part;
> @@ -796,6 +797,8 @@ static bool create_key_infos(const uchar *strpos, const 
> uchar *frm_image_end,
>   strpos+=7;
>}
>key_part->store_length=key_part->length;
> +  if(i)
> +  ext_key_length+= key_part->length;
>  }
>  
>  /*
> @@ -805,7 +808,8 @@ static bool create_key_infos(const uchar *strpos, const 
> uchar *frm_image_end,
>  keyinfo->ext_key_parts= keyinfo->user_defined_key_parts;
>  keyinfo->ext_key_flags= keyinfo->flags;
>  keyinfo->ext_key_part_map= 0;
> -if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME))
> +if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME)
> + && (pk_length + ext_key_length  <= 
> MAX_KEY_LENGTH))
>  {
>for (j= 0; 
> j < first_key_parts && keyinfo->ext_key_parts < MAX_REF_PARTS;
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] String functions have their item->maybe_null=true unnecessarily, blocking optimizations

[Sergey Petrunia]

Hi Alexander,

I was looking at 

https://jira.mariadb.org/browse/MDEV-11893

and found this piece of code:

bool Item_str_func::fix_fields(THD *thd, Item **ref)
{
  bool res= Item_func::fix_fields(thd, ref);
  /*
In Item_str_func::check_well_formed_result() we may set null_value
flag on the same condition as in test() below.
  */
  maybe_null= maybe_null || thd->is_strict_mode();
  return res;
}

This has been introduced by this cset:

https://github.com/MariaDB/server/commit/af22eb35e577ef17226faf662f2cffc4705bde26

Which says:

  Add Item_str_func::fix_fields() implementation, and set maybe_null to
  TRUE if we are in the SQL mode that requires some functions to return
  null even if they normally do not.

The patch has only one example of CHAR() function doing that.

At the same time, not having NOT-NULL attribute disallows query opimizations
(see MDEV-11893 for an example).
I think we should have this 

  maybe_null= maybe_null || thd->is_strict_mode();

logic on a case-by-case basis only.  I wanted to check with you - are there any
other known string functions that, as the patch puts it "return null even if 
they
normally do not"? 

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] b400696: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'

[Sergey Petrunia]

Hi Varun,

Please find my input below.

I would like again to voice my dissatisfaction that I have to point out coding
style violations in pretty much every patch I get.

The other error in the patch is also pretty basic, ok to have it once but I
think it is reasonable to expect more polished patches to be submitted for code
review.

On Sat, May 20, 2017 at 01:50:42AM +0530, Varun wrote:
> revision-id: b400696386529fa63e79f723bbb06c05f7bae820 
> (mariadb-10.2.2-416-gb400696)
> parent(s): fff61e31eca83a0a9af7c30e2bcda8309e9c695a
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-05-20 01:49:34 +0530
> message:
> 
> MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 
> 'key_buff'
> was corrupted, server crashes in opt_sum_query
> 
> extended keys feature disabled if the length of extended key is longer than 
> MAX_KEY_LEN
> 
> ---
>  mysql-test/r/innodb_ext_key.result | 45 
> ++
>  mysql-test/t/innodb_ext_key.test   | 31 ++
>  sql/table.cc   | 36 +-
>  3 files changed, 111 insertions(+), 1 deletion(-)
> 
> diff --git a/mysql-test/r/innodb_ext_key.result 
> b/mysql-test/r/innodb_ext_key.result
> index 1305be8..66391d6 100644
> --- a/mysql-test/r/innodb_ext_key.result
> +++ b/mysql-test/r/innodb_ext_key.result
> @@ -1133,5 +1133,50 @@ where index_date_updated= 10 and index_id < 800;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1SIMPLE  t2  range   index_date_updated  index_date_updated  
> 13  NULL#   Using index condition
>  drop table t0,t1,t2;
> +#
> +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 
> 'key_buff'
> +# was corrupted, server crashes in opt_sum_query
> +CREATE TABLE t1 (
> +pk INT,
> +f1 VARCHAR(3),
> +f2 VARCHAR(1024),
> +PRIMARY KEY (pk),
> +KEY(f2)
> +) ENGINE=InnoDB CHARSET utf8;
> +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def');
> +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk 
> <= 4;
> +MAX(t2.pk)
> +2
> +drop table t1;
> +CREATE TABLE t1 (
> +pk1 INT,
> +pk2 INT,
> +f1 VARCHAR(3),
> +f2 VARCHAR(1021),
> +PRIMARY KEY (pk1,pk2),
> +KEY(f2)
> +) ENGINE=InnoDB CHARSET utf8;
> +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def');
> +explain format= json
> +select * from t1 force index(f2)  where pk1 <= 5 and pk2 <=5 and f2 = 'abc' 
> and f1 <= '3';
> +EXPLAIN
> +{
> +  "query_block": {
> +"select_id": 1,
> +"table": {
> +  "table_name": "t1",
> +  "access_type": "range",
> +  "possible_keys": ["f2"],
> +  "key": "f2",
> +  "key_length": "3070",
> +  "used_key_parts": ["f2", "pk1"],
> +  "rows": 1,
> +  "filtered": 100,
> +  "index_condition": "t1.pk1 <= 5 and t1.pk2 <= 5 and t1.f2 = 'abc'",
> +  "attached_condition": "t1.f1 <= '3'"
> +}
> +  }
> +}
> +drop table t1;
>  set optimizer_switch=@save_ext_key_optimizer_switch;
>  SET SESSION STORAGE_ENGINE=DEFAULT;
...

> diff --git a/sql/table.cc b/sql/table.cc
> index 398383e..7d88eba 100644
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -2104,6 +2104,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, 
> bool write,
>  for (uint key=0 ; key < keys ; key++,keyinfo++)
>  {
>uint usable_parts= 0;
> +  KEY* key_first_info;
This variable may be used un-initialized.  If I change the above line to be 
  KEY* key_first_info=(KEY*)0x1;

and run this statement:

CREATE TABLE t1a (
  pk INT,
  f1 VARCHAR(3),
  f2 VARCHAR(1024),
  f2a VARCHAR(1024),
  PRIMARY KEY (pk),
  KEY(f2),
  KEY(f2a)
) ENGINE=InnoDB CHARSET utf8;

Then I get a crash.

>keyinfo->name=(char*) share->keynames.type_names[key];
>keyinfo->name_length= strlen(keyinfo->name);
>keyinfo->cache_name=
> @@ -2118,19 +2119,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, 
> bool write,
> keyinfo->name_length+1);
>}
>  
> +  if(!key)
Coding style: 'if (' ...

> +key_first_info= keyinfo;
> +
>if (ext_key_parts > share->key_parts && key)
>{
>  KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part +
>   (keyinfo-1)->ext_key_parts;
>  uint add_keyparts_for_this_key= add_first_key_parts;
> +uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0;
> +Field *field;
>  
>  /* 
>Do not extend the key that contains a component
>defined over the beginning of a field.
>   */ 
>  for (i= 0; i < keyinfo->user_defined_key_parts; i++)
> - {
> +{
Coding style: wrong identation.

>uint fieldnr= keyinfo->key_part[i].fieldnr;
> +  field= share->field[keyinfo->key_part[i].fieldnr-1];
> +
> +  if (field->null_ptr)
> +len_null_byte= HA_KEY_NULL_LENGTH;
> +
> +  if (field->type() == MYSQL_TYPE_BLOB ||
> + 

Re: [Maria-developers] [Commits] 5ac7ee3: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'

[Sergey Petrunia]

Hi Varun,

Good to see that the feedback from the last time was addressed.

There is another issue, though.

Consider a case where some (but not necessarily all) PK columns are present in
the secondary key:

create table t1 (
  pk1 VARCHAR(1000) not null,
  pk2 INT  not null,
  col2 INT not null,

  KEY key1(pk1, col2),
  PRIMARY KEY(pk1, pk2)
) CHARSET utf8 engine=innodb;
insert into t1 select a,a,a from ten;

In this case, key1 still has a suffix, but the suffix only includes columns
that are not already present in the key. 
That is, the "real" definition of key1 is

  (pk1, col2, pk2)

and not

  (pk1, col2, pk1, pk2)
.

Let's try that.

explain 
select * from t1 force index(key1) where pk1 < 'zz' and col2 <=2 and pk2 <=4;

Without the patch, I have all 3 key parts used:

| {
  "query_block": {
"select_id": 1,
"table": {
  "table_name": "t1",
  "access_type": "range",
  "possible_keys": ["key1"],
  "key": "key1",
  "key_length": "3010",
  "used_key_parts": ["pk1", "col2", "pk2"],
  "rows": 1,
  "filtered": 100,
  "attached_condition": "t1.pk1 <= '0' and t1.col2 <= 2 and t1.pk2 <= 4",
  "using_index": true
}
  }
} |

With the patch, I only see the explicitly defined key parts used:

| {
  "query_block": {
"select_id": 1,
"table": {
  "table_name": "t1",
  "access_type": "range",
  "possible_keys": ["key1"],
  "key": "key1",
  "key_length": "3006",
  "used_key_parts": ["pk1", "col2"],
  "rows": 1,
  "filtered": 100,
  "attached_condition": "t1.pk1 <= '0' and t1.col2 <= 2 and t1.pk2 <= 4",
  "using_index": true
}
  }
} |

Which I think is wrong, because the key length never exceeds MAX_KEY_LEN.

This can be easily fixed by appyling this patch over your patch:

--- table.cc.varun  2017-05-24 18:30:39.076341082 +0300
+++ table.cc2017-05-24 19:04:25.637740027 +0300
@@ -2164,10 +2164,13 @@
   for (i= 0; i < add_keyparts_for_this_key; i++)
   {
 uint pk_part_length= key_first_info->key_part[i].store_length;
-if (ext_key_length + pk_part_length > MAX_KEY_LENGTH)
+if (keyinfo->ext_key_part_map & 1< MAX_KEY_LENGTH)
+  {
+add_keyparts_for_this_key= i;
+break;
+  }
 }
 ext_key_length+= pk_part_length;
   }

Do you agree with this change? Any amendments? If it's ok, please add this
(together with the testcase) into your patch.


I've checked MySQL 5.7 also, and they do something weird:
...
insert into t1 select A.a + 1000*B.a, A.a + 1000*B.a, A.a + 1000*B.a from one_k 
A, ten B;
explain format=json
select * from t1 force index(key1) where pk1 <= '0' and col2 <=2 and pk2 <=4;
EXPLAIN
{
  "query_block": {
"select_id": 1,
"cost_info": {
  "query_cost": "1.41"
},
"table": {
  "table_name": "t1",
  "access_type": "range",
  "possible_keys": [
"key1"
  ],
  "key": "key1",
  "used_key_parts": [
"pk1"
  ],
  "key_length": "3010",
  "rows_examined_per_scan": 1,
  "rows_produced_per_join": 0,
  "filtered": "11.11",
  "using_index": true,
  "cost_info": {
"read_cost": "1.39",
"eval_cost": "0.02",
"prefix_cost": "1.41",
"data_read_per_join": "335"
  },
  "used_columns": [
"pk1",
"pk2",
"col2"
  ],
  "attached_condition": "((`test`.`t1`.`pk1` <= '0') and 
(`test`.`t1`.`col2` <= 2) and (`test`.`t1`.`pk2` <= 4))"
}
  }
}

key_length is 3010, but used_key_parts only includes "pk1"... The numbers do
not match.

On Tue, May 23, 2017 at 07:08:48PM +0530, Varun wrote:
> revision-id: 5ac7ee32a8e373d989bf9d665b5e01f770ac583f 
> (mariadb-10.1.20-282-g5ac7ee3)
> parent(s): a1b6128dedb4419db9fadaf94c356d3477d4e06f
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-05-23 19:00:15 +0530
> message:
> 
> MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 
> 'key_buff'
> was corrupted, server crashes in opt_sum_query
> 
> extended keys feature disabled if the length of extended key is longer than 
> MAX_KEY_LEN
> 
> ---
>  mysql-test/r/innodb_ext_key.result | 51 
> ++
>  mysql-test/t/innodb_ext_key.test   | 38 
>  sql/table.cc   | 36 ++-
>  3 files changed, 124 insertions(+), 1 deletion(-)
> 
> diff --git a/mysql-test/r/innodb_ext_key.result 
> b/mysql-test/r/innodb_ext_key.result
> index 1305be8..c76660c 100644
> --- a/mysql-test/r/innodb_ext_key.result
> +++ b/mysql-test/r/innodb_ext_key.result
> @@ -1133,5 +1133,56 @@ where index_date_updated= 10 and index_id < 800;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1SIMPLE  t2  range   index_date_updated  index_date_updated  
> 13  NULL#   Using index condition
>  drop

Re: [Maria-developers] [Commits] df94429: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'

[Sergey Petrunia]

Hi Varun,

Ok to push.

On Thu, May 25, 2017 at 07:41:50PM +0530, Varun wrote:
> revision-id: df944293721a7ad93d115a58f2b574067d1dc79f 
> (mariadb-10.1.20-282-gdf94429)
> parent(s): a1b6128dedb4419db9fadaf94c356d3477d4e06f
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-05-25 19:40:08 +0530
> message:
> 
> MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 
> 'key_buff'
> was corrupted, server crashes in opt_sum_query
> 
> extended keys feature disabled if the length of extended key is longer than 
> MAX_KEY_LEN
> 
> ---
>  mysql-test/r/innodb_ext_key.result | 81 
> ++
>  mysql-test/t/innodb_ext_key.test   | 51 
>  sql/table.cc   | 39 +-
>  3 files changed, 170 insertions(+), 1 deletion(-)
> 
> diff --git a/mysql-test/r/innodb_ext_key.result 
> b/mysql-test/r/innodb_ext_key.result
> index 1305be8..de1323e 100644
> --- a/mysql-test/r/innodb_ext_key.result
> +++ b/mysql-test/r/innodb_ext_key.result
> @@ -1133,5 +1133,86 @@ where index_date_updated= 10 and index_id < 800;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1SIMPLE  t2  range   index_date_updated  index_date_updated  
> 13  NULL#   Using index condition
>  drop table t0,t1,t2;
> +#
> +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 
> 'key_buff'
> +# was corrupted, server crashes in opt_sum_query
> +set @save_innodb_file_format= @@innodb_file_format;
> +set @save_innodb_large_prefix= @@innodb_large_prefix;
> +set global innodb_file_format = BARRACUDA;
> +set global innodb_large_prefix = ON;
> +CREATE TABLE t1 (
> +pk INT,
> +f1 VARCHAR(3),
> +f2 VARCHAR(1024),
> +PRIMARY KEY (pk),
> +KEY(f2)
> +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC;
> +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def');
> +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk 
> <= 4;
> +MAX(t2.pk)
> +2
> +drop table t1;
> +CREATE TABLE t1 (
> +pk1 INT,
> +pk2 INT,
> +f1 VARCHAR(3),
> +f2 VARCHAR(1021),
> +PRIMARY KEY (pk1,pk2),
> +KEY(f2)
> +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC;
> +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def');
> +explain format= json
> +select * from t1 force index(f2)  where pk1 <= 5 and pk2 <=5 and f2 = 'abc' 
> and f1 <= '3';
> +EXPLAIN
> +{
> +  "query_block": {
> +"select_id": 1,
> +"table": {
> +  "table_name": "t1",
> +  "access_type": "range",
> +  "possible_keys": ["f2"],
> +  "key": "f2",
> +  "key_length": "3070",
> +  "used_key_parts": ["f2", "pk1"],
> +  "rows": 1,
> +  "filtered": 100,
> +  "index_condition": "((t1.pk1 <= 5) and (t1.pk2 <= 5) and (t1.f2 = 
> 'abc'))",
> +  "attached_condition": "(t1.f1 <= '3')"
> +}
> +  }
> +}
> +drop table t1;
> +CREATE TABLE t1 (
> +f2 INT,
> +pk2 INT,
> +f1 VARCHAR(3),
> +pk1 VARCHAR(1000),
> +PRIMARY KEY (pk1,pk2),
> +KEY k1(pk1,f2)
> +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC;
> +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def');
> +explain format= json
> +select * from t1 force index(k1)  where f2 <= 5 and pk2 <=5 and pk1 = 'abc' 
> and f1 <= '3';
> +EXPLAIN
> +{
> +  "query_block": {
> +"select_id": 1,
> +"table": {
> +  "table_name": "t1",
> +  "access_type": "range",
> +  "possible_keys": ["k1"],
> +  "key": "k1",
> +  "key_length": "3011",
> +  "used_key_parts": ["pk1", "f2", "pk2"],
> +  "rows": 1,
> +  "filtered": 100,
> +  "index_condition": "((t1.f2 <= 5) and (t1.pk2 <= 5) and (t1.pk1 = 
> 'abc'))",
> +  "attached_condition": "(t1.f1 <= '3')"
> +}
> +  }
> +}
> +drop table t1;
>  set optimizer_switch=@save_ext_key_optimizer_switch;
> +set global innodb_file_format = @save_innodb_file_format;
> +set global innodb_large_prefix = @save_innodb_large_prefix;
>  SET SESSION STORAGE_ENGINE=DEFAULT;
> diff --git a/mysql-test/t/innodb_ext_key.test 
> b/mysql-test/t/innodb_ext_key.test
> index bf94b7d..c8acfc5 100644
> --- a/mysql-test/t/innodb_ext_key.test
> +++ b/mysql-test/t/innodb_ext_key.test
> @@ -778,5 +778,56 @@ where index_date_updated= 10 and index_id < 800;
>  
>  drop table t0,t1,t2;
>  
> +
> +--echo #
> +--echo # MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the 
> variable 'key_buff'
> +--echo # was corrupted, server crashes in opt_sum_query
> +
> +set @save_innodb_file_format= @@innodb_file_format;
> +set @save_innodb_large_prefix= @@innodb_large_prefix;
> +set global innodb_file_format = BARRACUDA;
> +set global innodb_large_prefix = ON;
> +
> +CREATE TABLE t1 (
> +  pk INT,
> +  f1 VARCHAR(3),
> +  f2 VARCHAR(1024),
> +  PRIMARY KEY (pk),
> +  KEY(f2)
> +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC;
> +
> +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def');
> +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk 
> <= 4;
> +drop table t1;
> +
> +CRE

Re: [Maria-developers] [Commits] 4d6b16f: Fixed the bug mdev-12838.

[Sergey Petrunia]

Hello Igor,

On Wed, May 31, 2017 at 10:45:07AM -0700, IgorBabaev wrote:
> revision-id: 4d6b16f10f9991821c128c382d32eb85248b3926 
> (mariadb-5.5.56-14-g4d6b16f)
> parent(s): b8405c853fa30002d164d5fe2b4f8ea8979c09b8
> author: Igor Babaev
> committer: Igor Babaev
> timestamp: 2017-05-31 10:45:07 -0700
> message:
> 
> Fixed the bug mdev-12838.
> 
...
> +bool JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join,
> + KEYUSE *keyuse)
> +{
> +  if (!access_from_tables_is_allowed(keyuse->used_tables, 
> + join->sjm_lookup_tables))
> +return false;
> +  if (join->sjm_scan_tables & table->map)
> +return true;
> +  table_map keyuse_sjm_scan_tables= keyuse->used_tables &
> +join->sjm_scan_tables;
> +  if (!keyuse_sjm_scan_tables)
> +return true;
> +  uint sjm_tab_nr= 0;
> +  while (!(keyuse_sjm_scan_tables & 1 << sjm_tab_nr))

The above "1" is a 32-bit number, while it should have the same bit length as
table_map (that is, 64). Please use "table_map(1)".


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 4d6b16f: Fixed the bug mdev-12838.

[Sergey Petrunia]

I can't find any other flaws in the patch.

I know we are not agreeing on which version should such patches be pushed into, 
let's discuss this on the call.

On Wed, Jun 07, 2017 at 05:49:35PM +0300, Sergey Petrunia wrote:
> Hello Igor,
> 
> On Wed, May 31, 2017 at 10:45:07AM -0700, IgorBabaev wrote:
> > revision-id: 4d6b16f10f9991821c128c382d32eb85248b3926 
> > (mariadb-5.5.56-14-g4d6b16f)
> > parent(s): b8405c853fa30002d164d5fe2b4f8ea8979c09b8
> > author: Igor Babaev
> > committer: Igor Babaev
> > timestamp: 2017-05-31 10:45:07 -0700
> > message:
> > 
> > Fixed the bug mdev-12838.
> > 
> ...
> > +bool JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join,
> > + KEYUSE *keyuse)
> > +{
> > +  if (!access_from_tables_is_allowed(keyuse->used_tables, 
> > + join->sjm_lookup_tables))
> > +return false;
> > +  if (join->sjm_scan_tables & table->map)
> > +return true;
> > +  table_map keyuse_sjm_scan_tables= keyuse->used_tables &
> > +join->sjm_scan_tables;
> > +  if (!keyuse_sjm_scan_tables)
> > +return true;
> > +  uint sjm_tab_nr= 0;
> > +  while (!(keyuse_sjm_scan_tables & 1 << sjm_tab_nr))
> 
> The above "1" is a 32-bit number, while it should have the same bit length as
> table_map (that is, 64). Please use "table_map(1)".
> 
> 
> BR
>  Sergei
> -- 
> Sergei Petrunia, Software Developer
> MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog
> 
> 

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Condition pushdown into derived tables and outer joins.

[Sergey Petrunia]

Hello Igor,

While reviewing MDEV-12845, I've found this piece in sql_select.cc:

while ((tbl= li++))
{
  /* 
Do not push conditions from where into materialized inner tables
of outer joins: this is not valid.
  */
  if (tbl->is_materialized_derived())
  {
/* 
  Do not push conditions from where into materialized inner tables
  of outer joins: this is not valid.
*/
if (!tbl->is_inner_table_of_outer_join())

Q1: Is int meaningful to have two identical comments?

Q2: It is not valid push parts of WHERE, but it is valid to push the parts of
ON expression.

An example:

explain format=json 
select * 
from 
  t21 LEFT JOIN 
  (select a,b, COUNT(*) as CNT from t22 group by a,b) TBL 
  ON (t21.a=TBL.a AND TBL.a<5);

here, "TBL.a<5" can be pushed down into the subquery, but it is not.

EXPLAIN EXTENDED confirms it:

  "query_block": {
"select_id": 1,
"const_condition": "1",
"table": {
  "table_name": "t21",
  "access_type": "ALL",
  "rows": 10,
  "filtered": 100
},
"table": {
  "table_name": "",
  "access_type": "ref",
  "possible_keys": ["key0"],
  "key": "key0",
  "key_length": "5",
  "used_key_parts": ["a"],
  "ref": ["test.t21.a"],
  "rows": 2,
  "filtered": 100,
  "attached_condition": "trigcond(trigcond(t21.a < 5 and t21.a is not 
null))",
  "materialized": {
"query_block": {
  "select_id": 2,
  "filesort": {
"sort_key": "t22.a, t22.b",
"temporary_table": {
  "table": {
"table_name": "t22",
"access_type": "ALL",
"rows": 10,
"filtered": 100
  }
}
  }
}
  }
}

Just in case, the commands to create the dataset were:

create table ten(a int);
insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);

create table t21 (a int, b int, c int);
insert into t21 select a,a,a from ten;

create table t22 (a int, b int, c int);
insert into t22 select a,a,a from ten;


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] ee73b33: MDEV-8840: ANALYZE FORMAT=JSON produces wrong data with BKA

[Sergey Petrunia]

Hi Varun,

On Sat, Jun 17, 2017 at 05:35:41PM +0530, Varun wrote:
> revision-id: ee73b331d97d33ae7aa1403a4ac7312f2d261754 
> (mariadb-10.1.20-323-gee73b33)
> parent(s): 056bab0880544d91ea67d18fe8db65b4f6625482
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-06-17 17:34:07 +0530
> message:
> 
> MDEV-8840: ANALYZE FORMAT=JSON produces wrong data with BKA
> 
> The issue was that r_loops, r_rows and r_filtered in ANALYZE FORMAT= JSON 
> were not
> calculated for the table on which we were performing the MRR scan in the BKA 
> join
> Fixed this by adding respective counter in the JOIN_CACHE_MRR::open and 
> JOIN_CACHE::next
> 

So I apply the patch and run this example (the dataset is from
explain_json.test, also pasted below):

MariaDB [j5]> analyze format=json select * from t3,t4 where t3.a=t4.a and 
(t4.c+1 < t3.b+1)\G
*** 1. row ***
ANALYZE: {
  "query_block": {
"select_id": 1,
"r_loops": 1,
"r_total_time_ms": 0.8859,
"table": {
  "table_name": "t3",
  "access_type": "ALL",
  "r_loops": 1,
  "rows": 10,
  "r_rows": 10,
  "r_total_time_ms": 0.0202,
  "filtered": 100,
  "r_filtered": 100,
  "attached_condition": "t3.a is not null"
},
"block-nl-join": {
  "table": {
"table_name": "t4",
"access_type": "ref",
"possible_keys": ["a"],
"key": "a",
"key_length": "5",
"used_key_parts": ["a"],
"ref": ["j5.t3.a"],
"r_loops": 1,
"rows": 1,
"r_rows": 11,
"r_total_time_ms": 0.0379,
"filtered": 100,
"r_filtered": 90.909
  },
  "buffer_type": "flat",
  "buffer_size": "256Kb",
  "join_type": "BKA",
  "mrr_type": "Rowid-ordered scan",
  "attached_condition": "t4.c + 1 < t3.b + 1",
  "r_filtered": 0
}
  }
}

I think, these value are wrong:
>"r_rows": 11,
>"r_filtered": 90.909

r_rows is actually 10,  r_filtered=100 (no attached condition)
I guess wrong r_filtered is caused by r_rows being wrong.


## Dataset:

set storage_engine=myisam;
set default_storage_engine=myisam;
create table t1(a int);
insert into t1 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
create table t2(a int);
insert into t2 select A.a + B.a* 10 + C.a * 100 from t1 A, t1 B, t1 C;
create table t3(a int, b int);
insert into t3 select a,a from t1;
create table t4(a int, b int, c int, filler char(100), key (a,b));
insert into t4 select a,a,a, 'filler-data' from t2;
set optimizer_switch='mrr=on';
set join_cache_level=6;


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] MDEV-12179: Per-engine mysql.gtid_slave_pos: auto-configuring/packaging

[Sergey Petrunia]

Hello,

Suppose there is a transactional storage engine that is shipped as a loadable 
module. The examples are MyRocks and TokuDB.

I think it the default behavior for such engine after MDEV-12179 should be that 
the engine is listed in @@gtid_pos_auto_engines. 

That is, as soon as a transaction that uses the storage engine is replicated,
mysql.slave_gtid_pos{_$engine} is created for it.

gtid_pos_auto_engines makes it easy to do this manually, but it's difficult 
to achieve as default behavior.  Here's an explanation why.

Loadable storage engines are put into their separate packages, for example
mariadb-plugin-rocksdb_10.2.7+maria-zesty_amd64.deb.

The two important files in the package are :

./etc/mysql/mariadb.conf.d/rocksdb.cnf
./usr/lib/mysql/plugin/ha_rocksdb.so

The first one is a .cnf that gets included into the server's cnf file. It 
would be nice to put there something like

  SET global gtid_pos_auto_engines= CONCAT(@@gtid_pos_auto_engines, 'rocksdb');

but that is not possible (it's a .cnf not .sql)

Possible solutions:

== [Ab]use the Storage Engine API ==

Make the storage engine put itself in @@gtid_pos_auto_engines. I think
this is technically possible, but
- It's a storage engine API abuse
- how does one explain @@gtid_pos_auto_engines to the users?

== Let gtid_pos_auto_engines include unknown engine ==

- Allow @@gtid_pos_auto_engines to list unknown engines
- set the default to include all known transactional storage engines, that is
it should be :

  gtid_pos_auto_engines='InnoDB,TokuDB,RocksDB'  // (*)

and the server should allow this even if it is built without TokuDB or RocksDB

(*) -- Aria is not needed as it is not a "true" transactional storage engine as 
far as interplay with the binlog is concerned.

== Let packaging use some other solution ==

I suppose it's possible for packaging system to run some script after a package
has been installed. that script might find/adjust my.cnf file to have the
desired @@gtid_pos_auto_engines setting.
This solution might be complex and fragile, though.

Any thoughts ? 


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 7801756: MDEV-13826: Floating point exception in Filesort_tracker::print_json_members(Json_writer*)

[Sergey Petrunia]

Hi Varun,

On Wed, Jul 12, 2017 at 12:43:02AM +0530, Varun wrote:
> revision-id: 780175646bb9312184d8a676e86b125f663b5156 
> (mariadb-10.2.2-591-g7801756)
> parent(s): 3904014ed37cc2b58844e8d24a5d6aaa69d80f59
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-07-12 00:41:52 +0530
> message:
> 
> MDEV-13826: Floating point exception in 
> Filesort_tracker::print_json_members(Json_writer*)
> 
> Whenever Filesort_tracker has r_loops=0, r_ouptut_rows would be 0, so we 
> should add the value zero to the member "r_output_rows" expliciytly
> 
> ---
>  mysql-test/r/analyze_format_json.result | 60 
> +
>  mysql-test/t/analyze_format_json.test   | 11 ++
>  sql/sql_analyze_stmt.cc | 11 +++---
>  3 files changed, 78 insertions(+), 4 deletions(-)
> 
> diff --git a/mysql-test/r/analyze_format_json.result 
> b/mysql-test/r/analyze_format_json.result
> index 686da92..436b13b 100644
> --- a/mysql-test/r/analyze_format_json.result
> +++ b/mysql-test/r/analyze_format_json.result
> @@ -757,3 +757,63 @@ ANALYZE
>}
>  }
>  drop table t1,t2,t3;
> +#
> +# MDEV-13286: Floating point exception in 
> Filesort_tracker::print_json_members(Json_writer*)
> +#
> +create table t0(a int);
> +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
> +create table t1 (a int, b int, c int);
> +insert into t1 select a,a,a from t0;
> +create table t2 as select * from t1;
> +analyze format=json select a, (select t2.b from t2 where t2.a t2.c limit 1) from t1 where t1.a<0;
> +ANALYZE
> +{
> +  "query_block": {
> +"select_id": 1,
> +"r_loops": 1,
> +"r_total_time_ms": 0.0324,
> +"table": {
> +  "table_name": "t1",
> +  "access_type": "ALL",
> +  "r_loops": 1,
> +  "rows": 10,
> +  "r_rows": 10,
> +  "r_total_time_ms": 0.0098,
> +  "filtered": 100,
> +  "r_filtered": 0,
> +  "attached_condition": "t1.a < 0"
> +},
> +"subqueries": [
> +  {
> +"expression_cache": {
> +  "state": "uninitialized",
> +  "r_loops": 0,
> +  "query_block": {
> +"select_id": 2,
> +"read_sorted_file": {
> +  "r_rows": null,
> +  "filesort": {
> +"sort_key": "t2.c",
> +"r_loops": 0,
> +"r_limit": "(varied across executions)",
> +"r_used_priority_queue": false,
Look at the "table" field below: the parameters whose values are not 
defined because of r_loop=0 had value of null (e.g. r_filtered, r_rows).

Let's follow that pattern here: r_loops:null, r_limit:null,
r_used_priority_queue: null.

Please change this.

Ok to push after this is addressed.
> +"r_output_rows": 0,
> +"table": {
> +  "table_name": "t2",
> +  "access_type": "ALL",
> +  "r_loops": 0,
> +  "rows": 10,
> +  "r_rows": null,
> +  "filtered": 100,
> +  "r_filtered": null,
> +  "attached_condition": "t2.a < t1.a"
> +}
> +  }
> +}
> +  }
> +}
> +  }
> +]
> +  }
> +}
> +drop table t0,t1,t2;
> diff --git a/mysql-test/t/analyze_format_json.test 
> b/mysql-test/t/analyze_format_json.test
> index 88a9077..02603bc 100644
> --- a/mysql-test/t/analyze_format_json.test
> +++ b/mysql-test/t/analyze_format_json.test
> @@ -212,3 +212,14 @@ GROUP BY sq ORDER BY gc;
>  
>  drop table t1,t2,t3;
>  
> +
> +--echo #
> +--echo # MDEV-13286: Floating point exception in 
> Filesort_tracker::print_json_members(Json_writer*)
> +--echo #
> +create table t0(a int);
> +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
> +create table t1 (a int, b int, c int);
> +insert into t1 select a,a,a from t0;
> +create table t2 as select * from t1;
> +analyze format=json select a, (select t2.b from t2 where t2.a t2.c limit 1) from t1 where t1.a<0;
> +drop table t0,t1,t2;
> diff --git a/sql/sql_analyze_stmt.cc b/sql/sql_analyze_stmt.cc
> index cc6e0f8..3e49af1 100644
> --- a/sql/sql_analyze_stmt.cc
> +++ b/sql/sql_analyze_stmt.cc
> @@ -43,15 +43,18 @@ void Filesort_tracker::print_json_members(Json_writer 
> *writer)
>}
>  
>writer->add_member("r_used_priority_queue"); 
> -  if (r_used_pq == get_r_loops())
> -writer->add_bool(true);
> -  else if (r_used_pq == 0)
> +  if (r_used_pq == 0)
>  writer->add_bool(false);
> +  else if (r_used_pq == get_r_loops())
> +writer->add_bool(true);
>else
>  writer->add_str(varied_str);
>  
> -  writer->add_member("r_output_rows").add_ll((longlong) rint(r_output_rows / 
> +  if (r_output_rows)
> +writer->add_member("r_output_rows").add_ll((longlong) rint(r_output_rows 
> /
>   get_r_loops()));
> +  else
> +writer->add_member("r_output_rows").add_ll((longlong)0);
>  
>if (sort_passes)
>{
> _

[Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks

[Sergey Petrunia]

Hello,

This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered 
an insteresting issue here, so I thought I would consult on both MyRocks and
MariaDB lists.

== Some background ==

"group commit with binlog" feature needs to accomplish two goals:

1. Keep the binlog and the storage engine in sync.
  This is done by employing XA between the binlog and the storage engine. It
  works by making these calls:
  
  /*
Make the transaction's changes to be ready to be committed (no conflicts
with other transactions, etc) but do not commit them yet.
The effects of the prepare operation must be synced to disk, as the storage 
engine needs to be able to recover (i.e. commit) the prepared transaction 
after a crash
  */
  storage_engine->prepare(sync=true);
  
  /* 
After this call, the transaction is considered committed. In case of a 
crash,
the recovery process will use the contents of the binlog to determine which
of the prepared transactions are to be committed and which are to be rolled
back.
  */
  binlog->write(sync=true);
  
  /*
Commit the transaction in the storage engine. This makes its changes 
visible 
to other transactions (and also releases its locks and so forth)
Note that most of the time(*) we don't need to sync there. In case of a 
crash
we will be able to recover using the binlog.
  */
  storage_engine->commit();

2. The second goal is to make operation performant.  We need two coordinated 
disk 
  flushes per transaction, the idea is to do "Group Commit" where multiple
  transactions share disk flushes.
  So, we need to do group commit and keep the storage engine and the binlog in
  sync while doing that.

== Group Commit with Binlog in MySQL ==

MySQL (and fb/mysql-5.6 in particular) does in the following phases:

Phase #1:
  Call storage_engine->prepare() for all transactions in the group.
  The call itself is not persistent.

Phase #2: Call storage->engine->flush_logs(). 
  This makes the effect of all Prepare operations from Phase#1 persistent.

Phase #3:
  Write and sync the binary log.

Phase #4: 
  Call storage_engine->commit(). This does not need to be persistent.

MyRocks implements them.


== Group Commit with Binlog in MariaDB ==

MariaDB does not have these phases described above:

>  Phase #1:
>Call storage_engine->prepare() for all transactions in the group.
>The call itself is not persistent.
>
>  Phase #2: Call storage->engine->flush_logs(). 
>This makes the effect of all Prepare operations from Phase#1 persistent.

A quote from Kristian's description at 
https://lists.launchpad.net/maria-developers/msg10832.html

>> So the idea is to do group prepare with the same group of transactions that
>> will later group commit to the binlog. In MariaDB, this concept does not
>> exist. Storage engine prepares are allowed to run in parallel and in any
>> order compared to binlog commit.

Initially this looked like it could work for MyRocks.

MyRocks has a group commit implementation, both Prepare() and Commit()
operations participate in groups.

However when I implemented a group commit implementation I found its
performance to be close to what one would expect if there was no commit 
grouping, and commit() call flushed to disk

https://jira.mariadb.org/browse/MDEV-11934 has the details.

== The issue ==

(I'm 95% certain about this. It's not 100% yet but it is very likely)

RocksDB's Group Write (see rocksdb/rocksdb/db/db_impl_write.cc,
DBImpl::WriteImpl function) handles both Prepare() and Commit() commands 
and does the following:

1. Controls writing the commited data into the MemTable
2. Writes transactions to WAL
3. Syncs the WAL.

All three steps are done for the whole group. This has a consequence: a
Commit() operation that does not need to sync the WAL will still be delayed
if another operation in the group needs the WAL to be synced.

This delay has a disastrous effect, because SQL layer tries to have the same 
order of transactions in the storage engine and in the binlog. In order to do 
that, it calls rocksdb_commit_ordered() for each transaction sequentially. 
Delaying one transaction causes a delay of the entire SQL-level commit group.


== Possible solutions ==

I am not sure what to do.

- Make the SQL layer's Group Commit implementation invoke hton->flush_logs()
  explicitly, like MySQL does?

- Modify RocksDB so that Transaction::Commit(sync=false) do not use Group
  Write? I am not sure if this is possible: Group Write is not about only 
  performance, it's about preventing concurrent MemTable writes.
  AFAIU one cannot just tell a certain DBImpl::WriteImpl() call to not
  participate in write groups and work as if there was no other activity.

- Modify RocksDB so that Transaction::Commit(sync=false) does not wait until
  its write group finishes WAL sync? This could be doable but is potentially
  complex.


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: serge

Re: [Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks

[Sergey Petrunia]

Hi Kristian,

On Mon, Sep 11, 2017 at 01:02:31PM +0200, Kristian Nielsen wrote:
> Sergey Petrunia  writes:
> 
> > This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered 
> > an insteresting issue here, so I thought I would consult on both MyRocks and
> > MariaDB lists.
> 
> Is your current code available somewhere?
> 

No, it wasn't. Just pushed the current code into bb-10.2-mdev11934 branch:

https://github.com/MariaDB/server/tree/bb-10.2-mdev11934

All changes are in the last commit.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] c132349bc79: MDEV-11846: ERROR 1114 (HY000) table full when performing GROUP BY

[Sergey Petrunia]

Hi Varun,

Ok to push.

On Fri, Sep 15, 2017 at 01:28:53PM +0530, Varun wrote:
> revision-id: c132349bc79d987947c8659e71601914cfec71e8 
> (mariadb-10.1.23-156-gc132349bc79)
> parent(s): fa2701c6f7b028782cf231565f578b2fc0f10d51
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-09-15 13:27:11 +0530
> message:
> 
> MDEV-11846: ERROR 1114 (HY000) table full when performing GROUP BY
> 
> The problem is there is an overflow for the key_file_length.
> Added the maximum limit for the key_file_length
> 
> ---
>  storage/maria/ma_create.c | 41 +
>  1 file changed, 33 insertions(+), 8 deletions(-)
> 
> diff --git a/storage/maria/ma_create.c b/storage/maria/ma_create.c
> index 0ddd8b226e2..79e60a76abc 100644
> --- a/storage/maria/ma_create.c
> +++ b/storage/maria/ma_create.c
> @@ -660,11 +660,24 @@ int maria_create(const char *name, enum data_file_type 
> datafile_type,
>  
>  if (length > max_key_length)
>max_key_length= length;
> -tot_length+= ((max_rows/(ulong) (((uint) maria_block_size -
> +
> +if (tot_length == ULLONG_MAX)
> +  continue;
> +
> +ulonglong tot_length_part= (max_rows/(ulong) (((uint) maria_block_size -
>MAX_KEYPAGE_HEADER_SIZE -
>KEYPAGE_CHECKSUM_SIZE)/
> - (length*2))) *
> -  maria_block_size);
> + (length*2)));
> +if (tot_length_part >=  (ULLONG_MAX / maria_block_size +
> +ULLONG_MAX % maria_block_size))
> +  tot_length= ULLONG_MAX;
> +else
> +{
> +  if (tot_length > ULLONG_MAX - tot_length_part * maria_block_size)
> +tot_length= ULLONG_MAX;
> +  else
> +tot_length+= tot_length_part * maria_block_size;
> +}
>}
>  
>unique_key_parts=0;
> @@ -673,11 +686,24 @@ int maria_create(const char *name, enum data_file_type 
> datafile_type,
>  uniquedef->key=keys+i;
>  unique_key_parts+=uniquedef->keysegs;
>  share.state.key_root[keys+i]= HA_OFFSET_ERROR;
> -tot_length+= (max_rows/(ulong) (((uint) maria_block_size -
> +
> +if (tot_length == ULLONG_MAX)
> +  continue;
> + ulonglong tot_length_part= (max_rows/(ulong) (((uint) maria_block_size -
>   MAX_KEYPAGE_HEADER_SIZE -
>   KEYPAGE_CHECKSUM_SIZE) /
> - ((MARIA_UNIQUE_HASH_LENGTH + pointer)*2)))*
> - (ulong) maria_block_size;
> + ((MARIA_UNIQUE_HASH_LENGTH + pointer)*2)));
> +
> +if (tot_length_part >=  (ULLONG_MAX / maria_block_size +
> +ULLONG_MAX % maria_block_size))
> +  tot_length= ULLONG_MAX;
> +else
> +{
> +  if (tot_length > ULLONG_MAX - tot_length_part * maria_block_size)
> +tot_length= ULLONG_MAX;
> +  else
> +tot_length+= tot_length_part * maria_block_size;
> +}
>}
>keys+=uniques; /* Each unique has 1 key */
>key_segs+=uniques; /* Each unique has 1 key seg */
> @@ -746,8 +772,7 @@ int maria_create(const char *name, enum data_file_type 
> datafile_type,
>  Get estimate for index file length (this may be wrong for FT keys)
>  This is used for pointers to other key pages.
>*/
> -  tmp= (tot_length + maria_block_size * keys *
> - MARIA_INDEX_BLOCK_MARGIN) / maria_block_size;
> +  tmp= (tot_length / maria_block_size + keys * MARIA_INDEX_BLOCK_MARGIN);
>  
>/*
>  use maximum of key_file_length we calculated and key_file_length value we
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] f87dddaaa95: MDEV-7865: Server crashes in Item_equal_iterator::get_curr_field on query

[Sergey Petrunia]

On Sat, Oct 28, 2017 at 12:47:03AM +0530, Varun wrote:
> revision-id: f87dddaaa95f9e7fc597c8839b539cc6f816c750 
> (mariadb-5.5.56-91-gf87dddaaa95)
> parent(s): fb5fe497e526f0677a7e2f86f34237e7efd4b806
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2017-10-28 00:46:43 +0530
> message:
> 
> MDEV-7865: Server crashes in Item_equal_iterator Item>::get_curr_field on query
>with impossible condition and OR/AND expressions
> 
> When multiple-equalites are merged into one in the function 
> Item_equal::merge_with_check, then the fields that are merged from one
> equality to the other still point to the previous mulitple_equality while the 
> should be pointing to the new equality
> after the merge is done
..
> diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc
> index ebe088e5092..8c79359ce88 100644
> --- a/sql/item_cmpfunc.cc
> +++ b/sql/item_cmpfunc.cc
> @@ -5832,7 +5832,18 @@ bool Item_equal::merge_with_check(Item_equal *item, 
> bool save_merged)
>if (intersected)
>{
>  if (!save_merged)
> +{
> +  Item *item_it;
> +  fi.rewind();
> +  while ((item_it= fi++))
> +  {
Wrong indentation

> +  if (!contains(fi.get_curr_field()))
> +  {
> +item_it->set_item_equal(this);

I am not sure if this can actually make a difference, but is there any reason
we don't just call set_item_equal unconditionally here?

My concern is that f.get_curr_field() checks for Field* objects, while 
set_item_equal() call applies to Items (either Item_field or 
Item_direct_view_ref)

What if there are two different Item objects that refer to the same field?
(this looks like it should not happen but I am not 100% certain about this).

> +  }
> +  }
>merge(item);
> +}
>  else
>  {
>Item *c= item->get_const();
> @@ -5843,9 +5854,12 @@ bool Item_equal::merge_with_check(Item_equal *item, 
> bool save_merged)
>  Item *item;
>  fi.rewind();
>  while ((item= fi++))
> - {
> +{
>if (!contains(fi.get_curr_field()))
> +  {
>  add(item);
> +item->set_item_equal(this);
> +  }

here, we have save_merged=true, that is, the old Item_equal will still be used.

Is this ok that we call set_item_equal($new_equality) for its members?

If an object is a participant in two Item_equal, which one should it point to?
the "most specific" one?  I'll try to ask Igor about this today.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] f87dddaaa95: MDEV-7865: Server crashes in Item_equal_iterator::get_curr_field on query

[Sergey Petrunia]

Discussed this on the optimizer call, please find the notes below.

On Wed, Nov 08, 2017 at 01:09:37PM +0300, Sergey Petrunia wrote:
> On Sat, Oct 28, 2017 at 12:47:03AM +0530, Varun wrote:
> > revision-id: f87dddaaa95f9e7fc597c8839b539cc6f816c750 
> > (mariadb-5.5.56-91-gf87dddaaa95)
> > parent(s): fb5fe497e526f0677a7e2f86f34237e7efd4b806
> > author: Varun Gupta
> > committer: Varun Gupta
> > timestamp: 2017-10-28 00:46:43 +0530
> > message:
> > 
> > MDEV-7865: Server crashes in Item_equal_iterator > Item>::get_curr_field on query
> >with impossible condition and OR/AND expressions
> > 
> > When multiple-equalites are merged into one in the function 
> > Item_equal::merge_with_check, then the fields that are merged from one
> > equality to the other still point to the previous mulitple_equality while 
> > the should be pointing to the new equality
> > after the merge is done
> ..
> > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc
> > index ebe088e5092..8c79359ce88 100644
> > --- a/sql/item_cmpfunc.cc
> > +++ b/sql/item_cmpfunc.cc
> > @@ -5832,7 +5832,18 @@ bool Item_equal::merge_with_check(Item_equal *item, 
> > bool save_merged)
> >if (intersected)
> >{
> >  if (!save_merged)
> > +{
> > +  Item *item_it;
> > +  fi.rewind();
> > +  while ((item_it= fi++))
> > +  {
> Wrong indentation
> 
> > +  if (!contains(fi.get_curr_field()))
> > +  {
> > +item_it->set_item_equal(this);
> 
> I am not sure if this can actually make a difference, but is there any reason
> we don't just call set_item_equal unconditionally here?
> 
> My concern is that f.get_curr_field() checks for Field* objects, while 
> set_item_equal() call applies to Items (either Item_field or 
> Item_direct_view_ref)
> 
> What if there are two different Item objects that refer to the same field?
> (this looks like it should not happen but I am not 100% certain about this).
>
There cannot be two Item_field objects that refer to the same field.

According to Igor, an Item_equal object cannot have both $item= Item_field
and an Item_*ref wich refers to the $item.

> > +  }
> > +  }
> >merge(item);
> > +}
> >  else
> >  {
> >Item *c= item->get_const();
> > @@ -5843,9 +5854,12 @@ bool Item_equal::merge_with_check(Item_equal *item, 
> > bool save_merged)
> >  Item *item;
> >  fi.rewind();
> >  while ((item= fi++))
> > -   {
> > +{
> >if (!contains(fi.get_curr_field()))
> > +  {
> >  add(item);
> > +item->set_item_equal(this);
> > +  }
> 
> here, we have save_merged=true, that is, the old Item_equal will still be 
> used.
> 
> Is this ok that we call set_item_equal($new_equality) for its members?
> 
> If an object is a participant in two Item_equal, which one should it point to?
> the "most specific" one?  I'll try to ask Igor about this today.

Asked, there is a concern about how item->item_equal is used.

In the worst case, we will need to do a deep copy, that is, clone the
Item_field (or Item_direct_view_ref) objects that participate in the
multiple-equalities.

(after writing this down, it struck me that if we create a new list of
Item_field objects for addition into the Item_equal, will we need to adjust
some pointers somewhere to point to the new objects? This seems to be more 
complex than the previous paragraph says)


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Handle failures from malloc

[Sergey Petrunia]

Hi Monty,

Please find my comments below. Anything not commented on seems to be ok to me.

> commit 7d9c24f6790c66d2f457837661852bf5e870b605
> Author: Michael Widenius 
> Date:   Tue Nov 14 07:47:58 2017 +0200
>
>   Handle failures from malloc
>   
>   Most "new" failures fixed in the following files:
>   - sql_select.cc
>   - item.cc
>   - opt_subselect.cc
>   


> diff --git a/sql/item.cc b/sql/item.cc
> index 2285ae28041..baae2a9b6fb 100644
> --- a/sql/item.cc
> +++ b/sql/item.cc
> @@ -1261,7 +1261,7 @@ Item *Item::safe_charset_converter(THD *thd, 
> CHARSET_INFO *tocs)
>if (!needs_charset_converter(tocs))
>  return this;
>Item_func_conv_charset *conv= new (thd->mem_root) 
> Item_func_conv_charset(thd, this, tocs, 1);
> -  return conv->safe ? conv : NULL;
> +  return conv && conv->safe ? conv : NULL;
>  }
>  
I am not fully convinced that returning NULL from this function is called
everywhere.
It is of course better to return NULL and hope for the best than to crash right
away.

> @@ -3296,6 +3301,8 @@ void Item_field::fix_after_pullout(st_select_lex 
> *new_parent, Item **ref,
>  }
>  
>  Name_resolution_context *ctx= new Name_resolution_context();
> +if (ctx)
> +  return;   // Fatal error set
>  if (context->select_lex == new_parent)
>  {
>/*

I assume this is a typo it should be "if (!ctx)"

> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index abbf2616537..7e42890143b 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -713,7 +713,7 @@ JOIN::prepare(TABLE_LIST *tables_init,
>union_part= unit_arg->is_unit_op();
>  
>if (select_lex->handle_derived(thd->lex, DT_PREPARE))
> -DBUG_RETURN(1);
> +DBUG_RETURN(-1);
>  
>thd->lex->current_select->context_analysis_place= NO_MATTER;
>thd->lex->current_select->is_item_list_lookup= 1;

I assume this is just for the sake of consistency with other DBUG_RETURN
statements in this function?

> @@ -14422,7 +14446,7 @@ static COND* substitute_for_best_equal_field(THD 
> *thd, JOIN_TAB *context_tab,
>  This works OK with PS/SP re-execution as changes are made to
>  the arguments of AND/OR items only
>*/
> -  if (new_item != item)
> +  if (new_item && new_item != item)
>  li.replace(new_item);
>  }

new_item was returned from a substitute_for_best_equal_field() call. That one
doesn't ever return NULL. It will return without substitution if it encounters
an error.
So this change is not needed.
 
> @@ -25044,8 +25103,11 @@ int JOIN::save_explain_data_intern(Explain_query 
> *output,
>  
>if (message)
>{
> -explain= new (output->mem_root) Explain_select(output->mem_root, 
> -   thd->lex->analyze_stmt);
> +if (!(explain= new (output->mem_root)
> +  Explain_select(output->mem_root,
> + thd->lex->analyze_stmt)))
> +  DBUG_RETURN(1);
> +
>  join->select_lex->set_explain_type(true);
>  
>  explain->select_id= join->select_lex->select_number;

The return value of this function is not checked.
I assume the above statement is there so that we dont try to continue to
execute the function if we've got an OOM error?

In this case, probably JOIN::save_explain_data_intern() should also have a
similar check for its 

  tab->save_explain_data(eta, used_tables, distinct_arg, first_top_tab);

call?
 

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 60d1803ec64: MDEV-6707: Wrong result (extra row) with group by, multi-part key

[Sergey Petrunia]

On Thu, Jan 11, 2018 at 02:35:55AM +0530, Varun wrote:
> revision-id: 60d1803ec645b6d388aac7125ac6381ecee4d548 
> (mariadb-5.5.56-115-g60d1803ec64)
> parent(s): 9c9cf556a15af84f9133ef859b50596085f7ae8e
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-01-11 01:51:23 +0530
> message:
> 
> MDEV-6707: Wrong result (extra row) with group by, multi-part key
> 
> Range optimizer creates ranges according to the where conditions. So this 
> case involves using a composite key,
> few parts of which are involved in GROUP BY and few in the MIN/MAX functions 
> in the select list.
> So during the execution of such queries, we try to find a prefix using the 
> fields involved in GROUP BY
> and we then check if this newly returned prefix lies within the range we had 
> calculated earlier.
> After we find a row that satisfies the partial range(involving fields of 
> GROUP BY), then we look at the other fields
> of the where clause. We look in all the ranges if we can satisfy the above 
> retured row.
> 
> The issue is while calculating these prefixes we might encounter same partial 
> ranges for multiple ranges.
>  An example would be
>  where condition is f2 ='c' AND f1 <> 9
>  so the ranges would be
>  (NULL,c <= f1,f2 <= c,9)
>  (c,9<= f1,f2 <= c,+infinity)
> 
>  In this case for calculating prefixes with the group by field we take up 
> the
>  partial ranges involving field f2 those would be
>  c<= f2 <=c
>  c<= f2 <=c
>  So we lookup rows with the same prefix in all such ranges and then we 
> check for the other
>  part(in this case f1) in all the ranges. So lets take a row of the table 
> to be (4,'c'), that would have
>  a prefix 'c' and value for f1 to be 4, so it would lie in the first 
> range mentioned above. But we would
>  get this same record again when we read the partial range and this would 
> again satisfy the first range.
> This issue can be fixed if we compare such partial ranges and don't lookup if 
> we see the same prefix again.

This is a huge and long comment. 
Please make it smaller, and move at least part of it into the code. Good places
would be: 
- The function comment for QUICK_RANGE_SELECT::get_next_prefix
- the new lines in QUICK_RANGE_SELECT::get_next_prefix which use/update the
  value of save_last_range

> 
> ---
>  mysql-test/r/range.result | 14 ++
>  mysql-test/r/range_mrr_icp.result | 14 ++
>  mysql-test/t/range.test   | 13 +
>  sql/opt_range.cc  | 20 +---
>  sql/opt_range.h   |  6 --
>  5 files changed, 62 insertions(+), 5 deletions(-)
> 
> diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result
> index 630a692cef6..72b31de4df4 100644
> --- a/mysql-test/r/range.result
> +++ b/mysql-test/r/range.result
> @@ -2144,3 +2144,17 @@ value1 1000685 12345
>  value1   1003560 12345
>  value1   1004807 12345
>  drop table t1;
> +#
> +# MDEV-6707: Wrong result (extra row) with group by, multi-part key
> +#
> +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB;
> +INSERT INTO t1 VALUES
> +(7,'v'),(0,'s'),(9,'l'),(4,'c');
> +explain
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  range   f2  f2  9   NULL2   Using 
> where; Using index for group-by
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +MAX(f1)  f2
> +4c
> +DROP TABLE t1;
> diff --git a/mysql-test/r/range_mrr_icp.result 
> b/mysql-test/r/range_mrr_icp.result
> index 3f5de5b0189..1050bfcd887 100644
> --- a/mysql-test/r/range_mrr_icp.result
> +++ b/mysql-test/r/range_mrr_icp.result
> @@ -2146,4 +2146,18 @@ value1 1000685 12345
>  value1   1003560 12345
>  value1   1004807 12345
>  drop table t1;
> +#
> +# MDEV-6707: Wrong result (extra row) with group by, multi-part key
> +#
> +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB;
> +INSERT INTO t1 VALUES
> +(7,'v'),(0,'s'),(9,'l'),(4,'c');
> +explain
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  range   f2  f2  9   NULL2   Using 
> where; Using index for group-by
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +MAX(f1)  f2
> +4c
> +DROP TABLE t1;
>  set optimizer_switch=@mrr_icp_extra_tmp;
> diff --git a/mysql-test/t/range.test b/mysql-test/t/range.test
> index 393ca68e945..62e907b7b4a 100644
> --- a/mysql-test/t/range.test
> +++ b/mysql-test/t/range.test
> @@ -1718,3 +1718,16 @@ where (key1varchar='value1' AND (key2int <=1 OR  
> key2int > 1));
>  --echo # The following must show col1=12345 for all rows:
>  select * from t1;
>  dr

Re: [Maria-developers] [Commits] a0ea3a58224: MDEV-14857: problem with 10.2.11 server crashing when executing stored procedure

[Sergey Petrunia]

Hi Sanja,

Please find my review below.
Lots of typos, but the two important comments are at the bottom, for the changes
in sql_select.cc and sql_trigger.cc

On Sat, Jan 27, 2018 at 11:39:10AM +0100, Oleksandr Byelkin wrote:
> revision-id: a0ea3a5822486fa1702d6615f1be30684c3d412f 
> (mariadb-10.1.30-26-ga0ea3a58224)
> parent(s): 524221e7a34d42214e82dd868348b453f2ef1591
> author: Oleksandr Byelkin
> committer: Oleksandr Byelkin
> timestamp: 2018-01-27 11:38:38 +0100
> message:
> 
> MDEV-14857: problem with 10.2.11 server crashing when executing stored 
> procedure
> 
> Counter for select numbering made stored with the statement (before was 
> global)
> So now it does have always accurate value which does not depend on
> interruption of statement prepare by errors like lack of table in
> a view definition.
> 
> ---

> diff --git a/sql/sql_class.h b/sql/sql_class.h
> index 4556487bdfe..2dc2fb61e9d 100644
> --- a/sql/sql_class.h
> +++ b/sql/sql_class.h
> @@ -1027,6 +1027,21 @@ class Statement: public ilink, public Query_arena
>  
>LEX_STRING name; /* name for named prepared statements */
>LEX *lex; // parse tree descriptor
> +  /*
> +LEX whith represent curent statement (conventional, SP or PS)
s/whith/which/ 
s/curent/current/
s/represent/represents/

> +
> +For example during view parsing THD::lex will point to the views LEX and
> +THD::stmt_lex will point on LEX of the statement where the view will be
s/point on/point to/
> +included
> +
> +Now used to have alwais correct select numbering inside statemet
s/Now used/Currently it is used/
s/alwais/always/
s/statemet/statement/

> +(LEX::current_select_number) without storing and restoring a global
> +counter which was THD::select_number.
> +
> +TODO: make some unified statement representation (now SP has different)
> +to store such data like LEX::current_select_number.
> +  */
> +  LEX *stmt_lex;
>/*
>  Points to the query associated with this statement. It's const, but
>  we need to declare it char * because all table handlers are written

> diff --git a/sql/sql_lex.h b/sql/sql_lex.h
> index b57fba08b47..a2da6f672ca 100644
> --- a/sql/sql_lex.h
> +++ b/sql/sql_lex.h
> @@ -727,7 +727,13 @@ class st_select_lex: public st_select_lex_node
>Item *prep_having;/* saved HAVING clause for prepared statement processing 
> */
>/* Saved values of the WHERE and HAVING clauses*/
>Item::cond_result cond_value, having_value;
> -  /* point on lex in which it was created, used in view subquery detection */
> +  /*
> + Point on lex in which it was created, used in view subquery detection.
"Points to the LEX...". Also please shift the whole comment 1 position to the 
left.

> +
> + TODO: make also st_select_lex::parent_stmt_lex (see THD::stmt_lex)
> + and use st_select_lex::parent_lex & st_select_lex::parent_stmt_lex
> + instead of global (from THD) references where it is possible.
> +  */
>LEX *parent_lex;
>enum olap_type olap;
>/* FROM clause - points to the beginning of the TABLE_LIST::next_local 
> list. */
> @@ -2452,7 +2458,7 @@ struct LEX: public Query_tables_list
>/** SELECT of CREATE VIEW statement */
>LEX_STRING create_view_select;
>  
> -  uint number_of_selects; // valid only for view
> +  uint current_select_number; // valid for statmet LEX (not view)
s/statmet/statement/
>  
>/** Start of 'ON table', in trigger statements.  */
>const char* raw_trg_on_table_name_begin;
> diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc
> index 6084c59a257..8d3116cd8b3 100644
> --- a/sql/sql_parse.cc
> +++ b/sql/sql_parse.cc
> @@ -6903,7 +6903,12 @@ void THD::reset_for_next_command(bool do_clear_error)
>  clear_error(1);
>  
>thd->free_list= 0;
> -  thd->select_number= 1;
> +  /*
> +The same thd->stmt_lex made in lex_start, but during bootstrup this
"We also assign thd->stmt_lex in lex_start()"

s/bootstrup/bootstrap

> +code executed first
code is executed

> +  */
> +  thd->stmt_lex= &main_lex; thd->stmt_lex->current_select_number= 1;
> +  DBUG_PRINT("info", ("Lex %p stmt_lex: %p", thd->lex, thd->stmt_lex));
>/*
>  Those two lines below are theoretically unneeded as
>  THD::cleanup_after_query() should take care of this already.

> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index b9fe8f3162a..7a6a028ee9c 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -2462,6 +2462,17 @@ void JOIN::save_explain_data(Explain_query *output, 
> bool can_overwrite,
>   bool need_tmp_table, bool need_order, 
>   bool distinct)
>  {
> +  /*
> +If there is SELECT in this statemet with the same number it must be the
> +same SELECT
> +  */
> +  DBUG_ASSERT(select_lex->select_number == UINT_MAX ||
> +  select_lex->select_number == INT_MAX ||
> +  !output ||

Is it really possible that output==NULL? If yes, that's probably 

Re: [Maria-developers] [Commits] f4f32ab71d5: MDEV-9744: session optimizer_use_condition_selectivity=5 causing SQL Error (1918):

[Sergey Petrunia]

Hi Varun,

On Sat, Feb 03, 2018 at 06:37:21PM +0530, Varun wrote:
> revision-id: f4f32ab71d5c96fbec87f8b6a5c057af2bc2398b 
> (mariadb-10.2.5-413-gf4f32ab71d5)
> parent(s): 9390ff53fc39e34976cf051ce33649f9316bb8e6
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-02-03 18:24:49 +0530
> message:
> 
> MDEV-9744: session optimizer_use_condition_selectivity=5 causing SQL Error 
> (1918):
>Encountered illegal value '' when converting to DECIMAL
> 
> The issue was that EITS data was allocated but then not read for some reason 
> (one being to avoid a deadlock),
> then the optimizer was using these bzero'ed buffers as EITS statistics.
> This should not be allowed, we should use statistcs for a table only when we 
> have successfully loaded/read
> the stats from the statistical tables.
> 
> ---
>  mysql-test/r/statistics.result | 16 
>  mysql-test/t/statistics.test   | 19 +++
>  sql/sql_statistics.cc  | 23 ---
>  3 files changed, 51 insertions(+), 7 deletions(-)
> 
> diff --git a/mysql-test/r/statistics.result b/mysql-test/r/statistics.result
> index ffaaf09acc8..1486a8ab5e6 100644
> --- a/mysql-test/r/statistics.result
> +++ b/mysql-test/r/statistics.result
> @@ -1716,5 +1716,21 @@ b
>  
> 0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.004,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.000,0.004,0.000,0.000,0.004,0.000,0.000,0.000,0.004,0.000,0.000,0.000,0.004,0.000,0.000,0.000,0.004,0.000,0.004,0.000,0.000,0.004,0.000,0.004,0.000,0.004,0.000,0.004,0.004,0.004,0.000,0.004,0.000,0.004,0.004,0.004,0.004,0.004,0.000,0.004,0.004,0.004,0.004,0.004,0.004,0.008,0.004,0.008,0.008,0.008,0.008,0.020,0.004,0.016,0.020,0.016,0.016,0.051,0.031,0.027,0.031,0.043,0.047,0.043,0.043,0.055,0.051,0.071,0.043,0.043,0.043,0.020,0.024,0.024,0.020,0.016,0.016,0.008,0.008,0.012,0.000
>  DROP TABLE t1;
>  #
> +# MDEB-9744: session optimizer_use_condition_selectivity=5 causing SQL Error 
> (1918): 
Please change to "MDEV"

> +# Encountered illegal value '' when converting to DECIMAL
> +#
> +set @save_optimizer_use_condition_selectivity= 
> @@optimizer_use_condition_selectivity;
> +set optimizer_use_condition_selectivity=3, use_stat_tables=preferably;
> +create table a (id int(10),cost decimal(9,2)) engine=innodb;
> +ANALYZE TABLE a PERSISTENT FOR ALL;
> +TableOp  Msg_typeMsg_text
> +test.a   analyze status  Engine-independent statistics collected
> +test.a   analyze status  OK
> +create temporary table b (id int);
> +insert into b (id) select id from a where cost > 0;
> +drop table a,b;
Please follow the convention and use table names t1,t2,...

Ok to push after this is addressed.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] a99dcdf7e9e: MDEV-6731: Valgrind warnings 'Invalid read' in subselect_engine::calc_const_tables with SQ in WHERE and

[Sergey Petrunia]

Hi Varun,

Please fix the MDEV# in the commit comment.
Ok to push after that.

On Wed, Mar 14, 2018 at 12:11:20PM +0530, Varun wrote:
> revision-id: a99dcdf7e9ede3b96466eb1294284141805747fc 
> (mariadb-10.0.30-308-ga99dcdf7e9e)
> parent(s): 48c11d407b409a1291c84782518c69ee9138ce72
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-03-14 12:09:49 +0530
> message:
> 
> MDEV-6731: Valgrind warnings 'Invalid read' in 
> subselect_engine::calc_const_tables with SQ in WHERE and
>HAVING, ORDER BY, materialization+semijoin
> 
> During cleanup a pointer to the materialised table that was freed was not set 
> to NULL
> 
> ---
>  mysql-test/r/having.result | 14 ++
>  mysql-test/t/having.test   | 18 ++
>  sql/sql_select.cc  |  4 +++-
>  3 files changed, 35 insertions(+), 1 deletion(-)
> 
> diff --git a/mysql-test/r/having.result b/mysql-test/r/having.result
> index 0b933427303..99059c25fbd 100644
> --- a/mysql-test/r/having.result
> +++ b/mysql-test/r/having.result
> @@ -723,4 +723,18 @@ SELECT * FROM t1 JOIN t2 ON c1 = c2 HAVING c2 > 'a' 
> ORDER BY c2 LIMIT 1;
>  c1   c2
>  xx
>  DROP TABLE t1,t2;
> +#
> +# MDEV-6736: Valgrind warnings 'Invalid read' in 
> subselect_engine::calc_const_tables with SQ 
> +# in WHERE and HAVING, ORDER BY, materialization+semijoin
> +#
> +CREATE TABLE t1 (a INT) ENGINE=MyISAM;
> +INSERT INTO t1 VALUES (3),(8);
> +CREATE TABLE t2 (b INT) ENGINE=MyISAM;
> +INSERT INTO t2 VALUES (2),(1);
> +SELECT a FROM t1
> +WHERE 9 IN ( SELECT MIN( a ) FROM t1 )  
> +HAVING a <> ( SELECT COUNT(*) FROM t2 ) 
> +ORDER BY a;
> +a
> +DROP TABLE t1,t2;
>  End of 10.0 tests
> diff --git a/mysql-test/t/having.test b/mysql-test/t/having.test
> index 1682fe5b874..e624cd630dd 100644
> --- a/mysql-test/t/having.test
> +++ b/mysql-test/t/having.test
> @@ -759,4 +759,22 @@ SELECT * FROM t1 JOIN t2 ON c1 = c2 HAVING c2 > 'a' 
> ORDER BY c2 LIMIT 1;
>  
>  DROP TABLE t1,t2;
>  
> +--echo #
> +--echo # MDEV-6736: Valgrind warnings 'Invalid read' in 
> subselect_engine::calc_const_tables with SQ 
> +--echo # in WHERE and HAVING, ORDER BY, materialization+semijoin
> +--echo #
> +
> +CREATE TABLE t1 (a INT) ENGINE=MyISAM;
> +INSERT INTO t1 VALUES (3),(8);
> + 
> +CREATE TABLE t2 (b INT) ENGINE=MyISAM;
> +INSERT INTO t2 VALUES (2),(1);
> + 
> +SELECT a FROM t1
> +WHERE 9 IN ( SELECT MIN( a ) FROM t1 )  
> +HAVING a <> ( SELECT COUNT(*) FROM t2 ) 
> +ORDER BY a;
> +
> +DROP TABLE t1,t2;
> +
>  --echo End of 10.0 tests
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index fd8ff6eb016..37d68c730dd 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -11476,13 +11476,15 @@ void JOIN_TAB::cleanup()
>}
>else
>{
> +TABLE_LIST *tmp= table->pos_in_table_list;
>  end_read_record(&read_record);
> -table->pos_in_table_list->jtbm_subselect->cleanup();
> +tmp->jtbm_subselect->cleanup();
>  /* 
>The above call freed the materializedd temptable. Set it to NULL so
>that we don't attempt to touch it if JOIN_TAB::cleanup() is invoked
>multiple times (it may be)
>  */
> +tmp->table= NULL;
>  table=NULL;
>}
>DBUG_VOID_RETURN;
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 0f35480f3d6: MDEV-6707: Wrong result (extra row) with group by, multi-part key

[Sergey Petrunia]

Hi Varun,

Nice to see feedback for the first review fixed, but there's more to do. Please
find more input below.

On Fri, Jan 19, 2018 at 01:07:20AM +0530, Varun wrote:
> revision-id: 0f35480f3d60fac91bf1f91f8140ac5f55724139 
> (mariadb-5.5.56-142-g0f35480f3d6)
> parent(s): fafdac3365f4943e73bcefd0e0d07d69997a9724
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-01-18 23:53:35 +0530
> message:
> 
> MDEV-6707: Wrong result (extra row) with group by, multi-part key
> 
> This case involves using a composite key,few parts of which are involved in 
> GROUP BY and few in the MIN/MAX
> functions in the select list.
> Ranges are created in accordance with the where condition, so during the 
> execution of such queries,
> we try to find a prefix using the fields involved in GROUP BY
> and we then check if this newly returned prefix lies within the range we had 
> calculated earlier.
> 
> For queries which use composite key, few parts of which are involved in GROUP 
> BY and few in the MIN/MAX functtions
> in the select list, we try to find the prefix of the ranges by using the 
> fields involved in the group by clause.
> We get extra rows in the output when we have same partial ranges created for 
> the fields in the GROUP BY clause
> 
> This issue can be fixed if we compare such partial ranges and don't lookup if 
> we see the same prefix again.
> 
> ---
>  mysql-test/r/range.result | 14 +
>  mysql-test/r/range_mrr_icp.result | 14 +
>  mysql-test/t/range.test   | 13 
>  sql/opt_range.cc  | 44 
> ---
>  sql/opt_range.h   |  5 +++--
>  5 files changed, 85 insertions(+), 5 deletions(-)
> 
> diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result
> index 630a692cef6..72b31de4df4 100644
> --- a/mysql-test/r/range.result
> +++ b/mysql-test/r/range.result
> @@ -2144,3 +2144,17 @@ value1 1000685 12345
>  value1   1003560 12345
>  value1   1004807 12345
>  drop table t1;
> +#
> +# MDEV-6707: Wrong result (extra row) with group by, multi-part key
> +#
> +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB;
> +INSERT INTO t1 VALUES
> +(7,'v'),(0,'s'),(9,'l'),(4,'c');
> +explain
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  range   f2  f2  9   NULL2   Using 
> where; Using index for group-by
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +MAX(f1)  f2
> +4c
> +DROP TABLE t1;
> diff --git a/mysql-test/r/range_mrr_icp.result 
> b/mysql-test/r/range_mrr_icp.result
> index 3f5de5b0189..1050bfcd887 100644
> --- a/mysql-test/r/range_mrr_icp.result
> +++ b/mysql-test/r/range_mrr_icp.result
> @@ -2146,4 +2146,18 @@ value1 1000685 12345
>  value1   1003560 12345
>  value1   1004807 12345
>  drop table t1;
> +#
> +# MDEV-6707: Wrong result (extra row) with group by, multi-part key
> +#
> +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB;
> +INSERT INTO t1 VALUES
> +(7,'v'),(0,'s'),(9,'l'),(4,'c');
> +explain
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  range   f2  f2  9   NULL2   Using 
> where; Using index for group-by
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +MAX(f1)  f2
> +4c
> +DROP TABLE t1;
>  set optimizer_switch=@mrr_icp_extra_tmp;
> diff --git a/mysql-test/t/range.test b/mysql-test/t/range.test
> index 393ca68e945..62e907b7b4a 100644
> --- a/mysql-test/t/range.test
> +++ b/mysql-test/t/range.test
> @@ -1718,3 +1718,16 @@ where (key1varchar='value1' AND (key2int <=1 OR  
> key2int > 1));
>  --echo # The following must show col1=12345 for all rows:
>  select * from t1;
>  drop table t1;
> +
> +--echo #
> +--echo # MDEV-6707: Wrong result (extra row) with group by, multi-part key
> +--echo #
> +
> +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB;
> +INSERT INTO t1 VALUES
> +(7,'v'),(0,'s'),(9,'l'),(4,'c');
> +
> +explain
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2;
> +DROP TABLE t1;
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index 25a9e729a8b..be959764b16 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -11249,6 +11249,7 @@ int QUICK_RANGE_SELECT::get_next()
>@param prefix_length   length of cur_prefix
>@param group_key_parts The number of key parts in the group prefix
>@param cur_prefix  prefix of a key to be searched for
> +  @param save_last_range INOUT Saving the last range we encountered
>  
>Each subsequent call to the method retrieves the first record that has 

Re: [Maria-developers] [Commits] b960f68b102: MDEV-11274: Executing EXPLAIN of complex query over join limit causes server to crash

[Sergey Petrunia]

Hi Varun,

On Tue, Jan 23, 2018 at 10:49:37AM +0530, Varun wrote:
> revision-id: b960f68b102ec72cf75252b357760e99c5dffc18 
> (mariadb-5.5.56-144-gb960f68b102)
> parent(s): a7a4519a40c58947796c6d9b2e4e58acc18aeef8
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-01-23 10:31:02 +0530
> message:
> 
> MDEV-11274: Executing EXPLAIN of complex query over join limit causes server 
> to crash
> 
> For this case we have a view that is mergeable but we are not able to merge 
> it in the
> parent select because that would exceed the maximum tables allowed in the 
> join list, so we
> materialise this view
> TABLE_LIST::dervied is NULL for such views, it is only set for views which 
> have ALGORITHM=TEMPTABLE
> Fixed by using SELECT_LEX_UNIT representing the result of the query within 
> the view definition
...

> diff --git a/sql/table.cc b/sql/table.cc
> index 9cade76cb78..8f377d5bda8 100644
> --- a/sql/table.cc
> +++ b/sql/table.cc
> @@ -6817,9 +6817,8 @@ int TABLE_LIST::fetch_number_of_rows()
>if (jtbm_subselect)
>  return 0;
>if (is_materialized_derived() && !fill_me)
> -
>{
> -table->file->stats.records= ((select_union*)derived->result)->records;
> +table->file->stats.records= ((select_union*)get_unit()->result)->records;
>  set_if_bigger(table->file->stats.records, 2);

So, regular VIEWs with algorithm=TEMPTABLE have 

  derived == view->unit

This is set in mysql_make_view():
...
table->derived_type= VIEW_ALGORITHM_TMPTABLE;
DBUG_PRINT("info", ("algorithm: TEMPORARY TABLE"));
view_select->linkage= DERIVED_TABLE_TYPE;
...

However, the view in the example has algorithm=MERGE so the execution doesn't
reach that point.
The decision to materialized is made in mysql_derived_merge. It calls 

  derived->set_materialized_derived();

which calls 
   
   set_check_materialized();

I have discussed this with Sanja, and the outcome is that a better solution
would be to set TABLE_LIST::derived when we decide that this view is going to
be handled with temp.table strategy.

set_check_materialized() seems to be the right place to do it (although its
name looks confusing. What does one "check"? Probably the name is a leftover 
from some feature backport from MySQL).


Could you change the patch to set TABLE_LIST::derived and see if that works?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] b718e87caaf: MDEV-11274: Executing EXPLAIN of complex query over join limit causes server to crash

[Sergey Petrunia]

Hi Varun,

On Sat, Mar 24, 2018 at 03:31:59AM +0530, Varun wrote:
> revision-id: b718e87caaf2d0a60c19cd042dfc04868f4bb0b4 
> (mariadb-5.5.56-187-gb718e87caaf)
> parent(s): dfc79ae949f6d831ff65a08a5d32d14b59ba06e6
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-03-24 03:31:18 +0530
> message:
> 
> MDEV-11274: Executing EXPLAIN of complex query over join limit causes server 
> to crash
> 
> For this case we have a view that is mergeable but we are not able to merge 
> it in the
> parent select because that would exceed the maximum tables allowed in the 
> join list, so we
> materialise this view
> TABLE_LIST::dervied is NULL for such views, it is only set for views which 
> have ALGORITHM=TEMPTABLE
> Fixed by making sure TABLE_LIST::derived is set for views that could not be 
> merged
> 
> ---
>  mysql-test/r/view.result | 197 
> +++
>  mysql-test/t/view.test   | 196 ++
>  sql/table.h  |   2 +
>  3 files changed, 395 insertions(+)
> 
> diff --git a/sql/table.h b/sql/table.h
> index 98f8c7ad73f..7b58f62e659 100644
> --- a/sql/table.h
> +++ b/sql/table.h
> @@ -46,6 +46,7 @@ struct TABLE_LIST;
>  class ACL_internal_schema_access;
>  class ACL_internal_table_access;
>  class Field;
> +struct LEX;
>  
>  /*
>Used to identify NESTED_JOIN structures within a join (applicable only to

What is the above needed for? Please remove this change.
Ok to push after this is addressed.

> @@ -2125,6 +2126,7 @@ struct TABLE_LIST
>  DBUG_PRINT("enter", ("Alias: '%s'  Unit: %p",
>  (alias ? alias : ""),
>   get_unit()));
> +derived= get_unit();
>  derived_type= ((derived_type & (derived ? DTYPE_MASK : DTYPE_VIEW)) |
> DTYPE_TABLE | DTYPE_MATERIALIZE);
>  set_check_materialized();

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 2950feb5c09: MDEV-9959: A serious MariaDB server performance bug

[Sergey Petrunia]

Hi Varun,

Please find my feedback below.

As disucssed before: please also run the MDEV's test case with this patch and
post a comment with the result.

On Fri, Mar 23, 2018 at 05:17:57PM +0530, Varun wrote:
> revision-id: 2950feb5c09033843a6a32c7431421b45d5812dd 
> (mariadb-10.3.0-646-g2950feb5c09)
> parent(s): 9ef7ab221d70630155206910de8a6053db097164
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-03-23 17:17:09 +0530
> message:
> 
> MDEV-9959: A serious MariaDB server performance bug
> 
> step#1: if a derived table has SELECT DISTINCT, provide index statistics for 
> it so that the join optimizer in the
> upper select knows that ref access to the table will produce one row.
> 
> Added handling for multiple selects in the derived table
> 
> ---
>  mysql-test/r/mdev9959.result | 64 
> 
>  mysql-test/t/mdev9959.test   | 28 +++
>  sql/sql_lex.h|  2 ++
>  sql/sql_union.cc | 36 +
>  sql/table.cc | 26 +++---
>  5 files changed, 153 insertions(+), 3 deletions(-)
> 
> diff --git a/mysql-test/r/mdev9959.result b/mysql-test/r/mdev9959.result
> new file mode 100644
> index 000..5269ba9b734
> --- /dev/null
> +++ b/mysql-test/r/mdev9959.result
> @@ -0,0 +1,64 @@
> +create table t1(a int);
> +insert into t1 values (1),(2),(3),(4),(5),(6);
> +create table t2(a int, b int,c int);
> +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5);
> +create table t3(a int, b int);
> +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2);
> +analyze select * from t1 , ( (select distinct t2.a from t2 group by c))q  
> where t1.a=q.a;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1PRIMARY t1  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using where
> +1PRIMARY   ref key0key05   test.t1.a   
> 1   0.83100.00  100.00  
> +2DERIVED t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using temporary; Using filesort

It is not at all clear which parts of the above output matter and which don't,
for example, does #rows matter for table t1?

Please add a comment (into .test file with --echo # ) describing that table
"" should have type=ref and rows=1 

> +select * from t1 , ( (select distinct t2.a from t2 group by c))q  where 
> t1.a=q.a;
> +aa
> +11
> +22
> +33
> +55
> +66
> +analyze select * from t1 , ( (select t2.a from t2 group by c))q  where 
> t1.a=q.a;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1PRIMARY t1  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using where
> +1PRIMARY   ref key0key05   test.t1.a   
> 2   0.83100.00  100.00  
> +2DERIVED t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using temporary; Using filesort
> +select * from t1 , ( (select t2.a from t2 group by c))q  where t1.a=q.a;
> +aa
> +11
> +22
> +33
> +55
> +66
> +analyze select * from t1 , ( (select distinct t2.a from t2 group by c) union 
> all (select distinct t2.a from t2 group by c))q  where t1.a=q.a;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1PRIMARY t1  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using where
> +1PRIMARY   ref key0key05   test.t1.a   
> 2   1.67100.00  100.00  
> +2DERIVED t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using temporary; Using filesort
> +3UNION   t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using temporary; Using filesort

What does the above test show?
We've got type=ref, #rows = 2 for the derived table.
The estimate is correct but where does it come from? 
As far as I understand this patch, it only sets the estimate rows=1 for derived
tables.  How do we arrive at the value of 2? 

(If we would have gotten rows=2 without this patch, what's the point of having
the testcase as part of this patch?)

> +select * from t1 , ( (select distinct t2.a from t2 group by c) union all 
> (select distinct t2.a from t2 group by c))q  where t1.a=q.a;
> +aa
> +11
> +11
> +22
> +22
> +33
> +33
> +55
> +55
> +66
> +66
> +analyze select * from t1 , ( (select distinct t2.a from t2 group by c) union 
> (select distinct t2.a from t2 group by c) union (select t3.a from t3 group by 
> b))q  where t1.a=q.a;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filter

Re: [Maria-developers] Please review a patch for MDEV-15758 Split Item_bool_func::get_mm_leaf() into virtual methods in Field and Type_handler

[Sergey Petrunia]

One question: what are SEL_ARG_LE/GE/GT classes? How do they relate to the base
SEL_ARG ?  If I have  "10 < t.key <= 20" which class it would be ?

If you are adding several new classes and an inheritance hierarchy, it would be
nice to have a comment describing the overall picture.

On Tue, Apr 03, 2018 at 12:57:22PM +0400, Alexander Barkov wrote:
> Hello Vicentiu,
> 
> Please review my patch for MDEV-15758.
> 
> It also fixes this bug:
> 
> MDEV-15759 Expect "Impossible WHERE" for
> indexed_int_column=out_of_range_int_constant
> 
> Thanks!

> diff --git a/mysql-test/main/type_bit.result b/mysql-test/main/type_bit.result
> index 30cd94c..c9796c8 100644
> --- a/mysql-test/main/type_bit.result
> +++ b/mysql-test/main/type_bit.result
> @@ -830,3 +830,21 @@ def  COALESCE(val, 
> 1)246 2   1   Y   32896   0   63
>  COALESCE(val, 1)
>  0
>  DROP TABLE t1;
> +#
> +# Start of 10.3 tests
> +#
> +#
> +# MDEV-15759 Expect "Impossible WHERE" for 
> indexed_int_column=out_of_range_int_constant
> +#
> +CREATE TABLE t1 (a BIT(7), KEY(a));
> +INSERT INTO t1 VALUES (1),(2),(3),(4),(5);
> +EXPLAIN SELECT * FROM t1 WHERE a=200;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  NULLNULLNULLNULLNULLNULLNULL
> Impossible WHERE noticed after reading const tables
> +EXPLAIN SELECT * FROM t1 WHERE a<=>200;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  NULLNULLNULLNULLNULLNULLNULL
> Impossible WHERE noticed after reading const tables
> +DROP TABLE t1;
> +#
> +# End of 10.3 tests
> +#
> diff --git a/mysql-test/main/type_bit.test b/mysql-test/main/type_bit.test
> index bb282fc..f663542 100644
> --- a/mysql-test/main/type_bit.test
> +++ b/mysql-test/main/type_bit.test
> @@ -458,3 +458,23 @@ DROP TABLE t2;
>  SELECT COALESCE(val, 1) FROM t1;
>  --disable_metadata
>  DROP TABLE t1;
> +
> +
> +--echo #
> +--echo # Start of 10.3 tests
> +--echo #
> +
> +--echo #
> +--echo # MDEV-15759 Expect "Impossible WHERE" for 
> indexed_int_column=out_of_range_int_constant
> +--echo #
> +
> +CREATE TABLE t1 (a BIT(7), KEY(a));
> +INSERT INTO t1 VALUES (1),(2),(3),(4),(5);
> +EXPLAIN SELECT * FROM t1 WHERE a=200;
> +EXPLAIN SELECT * FROM t1 WHERE a<=>200;
> +DROP TABLE t1;
> +
> +
> +--echo #
> +--echo # End of 10.3 tests
> +--echo #
> diff --git a/mysql-test/main/type_int.result b/mysql-test/main/type_int.result
> index 39e2e91..2125680 100644
> --- a/mysql-test/main/type_int.result
> +++ b/mysql-test/main/type_int.result
> @@ -93,3 +93,21 @@ DROP TABLE t1;
>  #
>  # End of 10.2 tests
>  #
> +#
> +# Start of 10.3 tests
> +#
> +#
> +# MDEV-15759 Expect "Impossible WHERE" for 
> indexed_int_column=out_of_range_int_constant
> +#
> +CREATE TABLE t1 (a TINYINT, KEY(a));
> +INSERT INTO t1 VALUES (1),(2),(3),(4),(5);
> +EXPLAIN SELECT * FROM t1 WHERE a=200;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  NULLNULLNULLNULLNULLNULLNULL
> Impossible WHERE noticed after reading const tables
> +EXPLAIN SELECT * FROM t1 WHERE a<=>200;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  NULLNULLNULLNULLNULLNULLNULL
> Impossible WHERE noticed after reading const tables
> +DROP TABLE t1;
> +#
> +# End of 10.3 tests
> +#
> diff --git a/mysql-test/main/type_int.test b/mysql-test/main/type_int.test
> index 271b4d5..899af44 100644
> --- a/mysql-test/main/type_int.test
> +++ b/mysql-test/main/type_int.test
> @@ -76,3 +76,22 @@ DROP TABLE t1;
>  --echo #
>  --echo # End of 10.2 tests
>  --echo #
> +
> +--echo #
> +--echo # Start of 10.3 tests
> +--echo #
> +
> +--echo #
> +--echo # MDEV-15759 Expect "Impossible WHERE" for 
> indexed_int_column=out_of_range_int_constant
> +--echo #
> +
> +CREATE TABLE t1 (a TINYINT, KEY(a));
> +INSERT INTO t1 VALUES (1),(2),(3),(4),(5);
> +EXPLAIN SELECT * FROM t1 WHERE a=200;
> +EXPLAIN SELECT * FROM t1 WHERE a<=>200;
> +DROP TABLE t1;
> +
> +
> +--echo #
> +--echo # End of 10.3 tests
> +--echo #
> diff --git a/mysql-test/main/update.result b/mysql-test/main/update.result
> index 73ebb73..ccf2f44 100644
> --- a/mysql-test/main/update.result
> +++ b/mysql-test/main/update.result
> @@ -447,7 +447,7 @@ UPDATE t1 SET user_id=null WHERE request_id=9;
>  show status like '%Handler_read%';
>  Variable_nameValue
>  Handler_read_first   0
> -Handler_read_key 3
> +Handler_read_key 2
>  Handler_read_last0
>  Handler_read_next0
>  Handler_read_prev0
> @@ -459,7 +459,7 @@ UPDATE t1 SET user_id=null WHERE 
> request_id=99;
>  show status like '%Handler_read%';
>  Variable_nameValue
>  Handler_read_first   0
> -Handler_read_key 3
> +Handler_read_key   

Re: [Maria-developers] [Commits] a74e3ef17e7: MDEV-14551 Can't find record in table on multi-table update with ORDER BY

[Sergey Petrunia]

Hi Serg,

Please find some cosmetic input below.

(additionally, we've found a failing example while chatting)

On Tue, Apr 10, 2018 at 02:10:17PM +0200, s...@mariadb.org wrote:
> revision-id: a74e3ef17e7a8693d68001290a8d91b5b206804d 
> (mariadb-10.3.5-138-ga74e3ef17e7)
> parent(s): 9bd3af97dffa411657879bbdfd4dfef38c99f7cc
> author: Sergei Golubchik
> committer: Sergei Golubchik
> timestamp: 2018-04-10 14:09:17 +0200
> message:
> 
> MDEV-14551 Can't find record in table on multi-table update with ORDER BY

...
> diff --git a/mysql-test/main/multi_update.test 
> b/mysql-test/main/multi_update.test
> index 5feebe87a5a..96e78dab82b 100644
> --- a/mysql-test/main/multi_update.test
> +++ b/mysql-test/main/multi_update.test
> @@ -914,3 +914,14 @@ update t1 set c1=NULL;
>  update t1, t2 set t1.c1=t2.c3 where t1.c3=t2.c3 order by t1.c3 desc limit 2;
>  select * from t1;
>  drop table t1, t2;
> +
> +#
> +# MDEV-14551 Can't find record in table on multi-table update with ORDER BY
> +#
> +CREATE TABLE t1 (i INT) ENGINE=MEMORY;
> +INSERT t1 VALUES (1),(2);
> +CREATE TABLE t2 (f INT) ENGINE=MyISAM;
> +INSERT t2 VALUES (1),(2);
> +UPDATE t1, t2 SET f = 126 ORDER BY f LIMIT 2;
> +SELECT * FROM t2;
> +DROP TABLE t1, t2;
> diff --git a/sql/item.h b/sql/item.h
> index 9574bdc63bf..e391e7810c4 100644
> --- a/sql/item.h
> +++ b/sql/item.h
> @@ -2023,6 +2023,7 @@ class Item: public Value_source,
>{
>  marker &= ~EXTRACTION_MASK;
>}
> +  virtual TABLE *rowid_table() const { return 0; }

I am concerned about this approach bloating class Item's vtable. 

There is (and likely will ever be) only one class with a different 
implementation, the check for this is made only in one place - why not
check Item's type() or functype(), etc?

>  };
>  
>  MEM_ROOT *get_thd_memroot(THD *thd);
> diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc
> index c6b0ae7fba8..9097ffca867 100644
> --- a/sql/item_strfunc.cc
> +++ b/sql/item_strfunc.cc
> @@ -5204,3 +5204,23 @@ String *Item_func_dyncol_list::val_str(String *str)
>  my_free(names);
>return NULL;
>  }
> +
> +Item_temptable_rowid::Item_temptable_rowid(TABLE *table_arg)
> +  : Item_str_func(table_arg->in_use), table(table_arg)
> +{
> +  max_length= table->file->ref_length;
> +}
> +
> +void Item_temptable_rowid::fix_length_and_dec()
> +{
> +  used_tables_cache= table->map;
> +  const_item_cache= false;
> +}
> +
> +String *Item_temptable_rowid::val_str(String *str)
> +{
> +  if (!((null_value= table->null_row)))
> +table->file->position(table->record[0]);
> +  str_value.set((char*)(table->file->ref), max_length, &my_charset_bin);
> +  return &str_value;
> +}
> diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h
> index 18cda491efd..49de5568696 100644
> --- a/sql/item_strfunc.h
> +++ b/sql/item_strfunc.h
> @@ -1748,5 +1748,20 @@ class Item_func_dyncol_list: public Item_str_func
>{ return get_item_copy(thd, this); }
>  };
>  
> -#endif /* ITEM_STRFUNC_INCLUDED */

Please add a note that this is not the "_rowid" that we support in the parser.

> +class Item_temptable_rowid :public Item_str_func
> +{
> +  TABLE *table;
> +public:
> +  Item_temptable_rowid(TABLE *table_arg);
> +  const Type_handler *type_handler() const { return &type_handler_string; }
> +  Field *create_tmp_field(bool group, TABLE *table)
> +  { return create_table_field_from_handler(table); }
> +  String *val_str(String *str);
> +  const char *func_name() const { return ""; }
> +  void fix_length_and_dec();
> +  Item *get_copy(THD *thd)
> +  { return get_item_copy(thd, this); }
> +  TABLE *rowid_table() const { return table; }
> +};
>  
> +#endif /* ITEM_STRFUNC_INCLUDED */
...

> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 796ea569e64..22301319680 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -2650,6 +2650,21 @@ bool JOIN::add_having_as_table_cond(JOIN_TAB *tab)
>  }
>  
>  
> +bool JOIN::add_fields_for_current_rowid(JOIN_TAB *cur, List 
> *table_fields)

I think ths is actually "current rowids", in plural. As the loop shows, there
are multiple current rowids (from different tables).

> +{
> +  for (JOIN_TAB *tab=join_tab; tab < cur; tab++)

Please add a comment:
this will not walk into semi-join materialization nests but this is ok 
because we will never need to save current rowids for those.

> +  {
> +if (!tab->keep_current_rowid)
> +  continue;
> +Item *item= new (thd->mem_root) Item_temptable_rowid(tab->table);
> +item->fix_fields(thd, 0);
> +table_fields->push_back(item, thd->mem_root);
> +cur->tmp_table_param->func_count++;
> +  }
> +  return 0;
> +}
> +
> +
>  /**
>Set info for aggregation tables
>  
> diff --git a/sql/sql_select.h b/sql/sql_select.h
> index f8911fbba01..1da87bb9d50 100644
> --- a/sql/sql_select.h
> +++ b/sql/sql_select.h
> @@ -1438,6 +1438,9 @@ class JOIN :public Sql_alloc
>
>enum { QEP_NOT_PRESENT_YET, QEP_AVAILABLE, QEP_DELETED} have_query_plan;
>  
> +  // if keep_current_rowid=true, whether they should

Re: [Maria-developers] a74e3ef17e7: MDEV-14551 Can't find record in table on multi-table update with ORDER BY

[Sergey Petrunia]

On Thu, Apr 12, 2018 at 07:22:56PM +0200, Sergei Golubchik wrote:
> Hi, Sergey!
> 
> On Apr 12, Sergey Petrunia wrote:
> > > diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h
> > > index 18cda491efd..49de5568696 100644
> > > --- a/sql/item_strfunc.h
> > > +++ b/sql/item_strfunc.h
> > > @@ -1748,5 +1748,20 @@ class Item_func_dyncol_list: public Item_str_func
> > >{ return get_item_copy(thd, this); }
> > >  };
> > >  
> > > -#endif /* ITEM_STRFUNC_INCLUDED */
> > 
> > Please add a note that this is not the "_rowid" that we support in the 
> > parser.
> 
> I used the terminology that you introduced, that is
> "keep_current_rowid".
> 
> But if you don't mind, I'd rather rename keep_current_rowid (and
> every "rowid" that I introduced) to keep_current_position (because it's
> handler::position()) or keep_current_ref (because it's handler::ref and
> handler::ref_length), whichever is less ambiguous.

I didn't want to request any renames actually. We use the term "rowid" in many
places in the code.
Until now, there has been no intersection with the SQL-level "_rowid".  Now,
with the new Item-derived class I was afraid there would be some confusion and 
wanted a comment to prevent it, that's all.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] a74e3ef17e7: MDEV-14551 Can't find record in table on multi-table update with ORDER BY

[Sergey Petrunia]

On Thu, Apr 12, 2018 at 07:45:37PM +0200, Sergei Golubchik wrote:
> Hi, Sergey!
> 
> Forgot to mention it in my first reply, sorry
> 
> On Apr 12, Sergey Petrunia wrote:
> > > diff --git a/sql/item.h b/sql/item.h
> > > index 9574bdc63bf..e391e7810c4 100644
> > > --- a/sql/item.h
> > > +++ b/sql/item.h
> > > @@ -2023,6 +2023,7 @@ class Item: public Value_source,
> > >{
> > >  marker &= ~EXTRACTION_MASK;
> > >}
> > > +  virtual TABLE *rowid_table() const { return 0; }
> > 
> > I am concerned about this approach bloating class Item's vtable. 
> > 
> > There is (and likely will ever be) only one class with a different 
> > implementation, the check for this is made only in one place - why not
> > check Item's type() or functype(), etc?
> 
> I know, I don't like this either. But the alternative was
> 
>if (item->type() == FUNC_ITEM &&
>((Item_func*)item)->functype() == ROWID_FUNC &&
>((Item_temptable_rowid*)item)->table == tbl)
> ...
> 
> that is, I need to compare both item->type() and item_func->functype().
> So I thought it'd be better to add a method.
> 
> I can revert back to type()/functype() comparison, if you prefer that.
>
I would prefer that, if several checks are needed a short static/inline
function will be more readable...

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] : GSoC Project Optimizer Trace

[Sergey Petrunia]

Hi Zhzhzoo,

Please find below some thoughts about the optimizer trace project

* The MDEV for the optimizer trace task is
https://jira.mariadb.org/browse/MDEV-6111. Let's keep it up to date with all
the changes.

* I think we should target MySQL-like functionality of optimizer_trace. At
least, for the first milestone. We may decide to make some things better 
but working in a similar way to MySQL is a nice first goal.

* MariaDB's optimizer trace will contain JSON and produce similar content to
MySQL. However, MariaDB's optimizer is different, so one cannot expect that
the JSON document will have exactly the same fields/objects/etc. (MariaDB' 
EXPLAIN FORMAT=JSON output doesn't match the one in MySQL, either. And in 
MySQL, the output varies across its versions.)

* MySQL prints conditions with heavy over-quoting.
Example:

 "original_condition": "((`A`.`a` = `B`.`a`) and ((`A`.`a` + `B`.`a`) < 10))",

MariaDB has the code to print the condition without redundant quotes (see
how EXPLAIN FORMAT=JSON does it).

* I assume MySQL has some class for printing well-formed JSON documents.
MariaDB also has one. It is in sql/my_json_writer.h, see class Json_writer.
It handles printing into JSON, and also does pretty-printing. When I looked at
the code last time, it did pretty-printing better than MySQL's code for EXPLAIN
FORMAT=JSON did.
It would be nice to use Json_writer for collecting optimizer trace output, too.
(there is a caveat: Json_writer was written specifically for producing 
EXPLAIN FORMAT=JSON output so it may lack some features. You may need to 
extend it).

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] MDEV-6111: Optimizer trace: features and milestones

[Sergey Petrunia]

Hi Zhzhzoo,

Please find below some thoughts about features/milestones in the project. Items
at the top of the list should be done first.

The list is not necesarrily complete, any comments/additions are welcome.

- Optimizer trace collection API (is it different from Json_writer that 
  we have? Does MySQL use the same API to produce EXPLAIN FORMAT=JSON
  and produce optimizer_trace? Should we follow them?)

- Basic optimizer trace collection (just print the query and 
  "hello":"world" instead of trace). Clear the trace data at query start,
  collect the trace if tracing is switched on (and do not collect if it is off)

- INFORMATION_SCHEMA.OPTIMIZER_TRACE table in I_S database.

  Produce optimizer_trace content for features common with MySQL:
  = condition rewrite
  = partition pruning
  = range analysis
  = ref/eq_ref analysis (update_ref_and_keys)
  = (TODO what else?)

  Produce optimizer_trace content for MariaDB-specific features:
  = merged semi-join conversion
  = non-merged semi-join conversion
  = join condition pushdown
  = (TODO what else?)

- optimizer_trace_limit/optimizer_trace_offset (without this feature, 
  just save one last trace)

- Ability to produce either a pretty-printed or one-line optimizer trace

- Security
  = INFORMATION_SCHEMA.OPTIMIZER_TRACE.INSUFFICIENT_PRIVILEGES
  = Who can view optimizer_trace?
  = Figure out and implement the necessary security restrictions

- Memory limits
  = Honor the @@optimizer_trace_max_mem_size setting.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] de722585c1b: MDEV-16225: wrong resultset from query with semijoin=on

[Sergey Petrunia]

Hi Varun,


On Thu, May 24, 2018 at 03:17:02AM +0530, Varun wrote:
> revision-id: de722585c1b2ad2af99bd1749e8566ea77838764 
> (mariadb-5.5.56-219-gde722585c1b)
> parent(s): 1ada4afb0a51f7283b6187a95019ec2cb80c8a0b
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-05-24 03:09:42 +0530
> message:
> 
> MDEV-16225: wrong resultset from query with semijoin=on
> 
> For non-semi-join subquery optimization we do a cost based decision between
> Materialisation and IN -> EXIST transformation. The issue in this case is 
> that for IN->EXIST transformation
> we run JOIN::reoptimize with the IN->EXISt conditions and we come up with a 
> new query plan. But when we compare
> the cost with Materialization, we make the decision to chose Materialization 
> so we need to restore the query plan
> for Materilization. The restoring of query plan is not handled correctly.
> 
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index d6d269a700f..b8d43b9797e 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -23805,6 +23805,22 @@ void JOIN::restore_query_plan(Join_plan_state 
> *restore_from)
>  }
>  
>}
> +  else
> +  {
> +/*
> +  After running JOIN::reoptimize for IN->EXISTS
> +  transformation we might add elements to the keyuse array,
> +  The original query plan was that the keyuse array had no elements, so 
> we
> +  should restore that state here.
> +*/
> +while(keyuse.elements)
> +  delete_dynamic_element(&keyuse, 0);

I'm wondering, why do we need the above loop and not just delete everything
with delete_dynamic().

But if I don't miss anything, this is not significant.
Ok to push either the above variant or one with delete_dynamic().

> +for (uint i= 0; i < table_count; i++)
> +{
> +  join_tab[i].keyuse= NULL;
> +  join_tab[i].checked_keys.clear_all();
> +}
> +  }
>memcpy((uchar*) best_positions, (uchar*) restore_from->best_positions,
>   sizeof(POSITION) * (table_count + 1));
>/* Restore SJM nests */
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Conservative parallel slave is "too optimistic" for certain engines

[Sergey Petrunia]

Hello,

This comes from MDEV-16242.

== Symptoms == 
When one runs a parallel slave (mode=conservative) and replicates DML for 
MyRocks table without a Primary Key, replication may stop with a
ER_KEY_NOT_FOUND error.

This may happen even if the queries were run on the master sequentially.

== A detail about conservative replication ==

Suppose the master runs these two statements, sequentially:

INSERT INTO t1 VALUES (5),(6),(7);
DELETE FROM t1;

Parallel slave may schedule the INSERT to Thread1, and the DELETE to Thread2.
In conservative parallel replication, the DELETE "will wait" to be executed
after the INSERT.

One may expect that "will wait" here means that DELETE execution does not
start until the INSERT has committed. But it's more subtle than that: actually,
DELETE execution will start as soon as the INSERT is ready to do the 
prepare/commit steps.

I assume this was done to increase the parallelism: initial phases of DELETE
can be ran in parallel with commit/prepare steps of the INSERT. This is safe to
do:

1. The INSERT has acquired locks for all rows it is about to touch. DELETE
will not be able to prevent INSERT from committing.

2. The DELETE starts its execution on a database that doesn't include the
results of the INSERT (the INSERT has not committed yet!). But this is fine, 
because the DELETE locks the rows it is about to modify. If it attempts to 
access a row that the INSERT is about to insert, it will block on a lock 
until the INSERT finishes.


== Applying this to MyRocks ==

The critical part is #2. It works if the DELETE command will acquire row locks
for rows it about to DELETE. This is normally true:

If the storage engine supports gap locks, attempting to read a locked gap
will cause DELETE to wait for INSERT

If the storage engine doesn't support gap locks, it will use 
Row-Based-Replication. 
RBR's Delete_rows_event includes Primary Key value of the row that should be 
deleted. Doing a point lookup on PK will cause the DELETE to block on the row 
lock.

The remaining case is
- storage engine that doesn't support gap locks
- Row-based-replication of table without primary key

In this case, finding the row to delete is done by scanning the table until we
find the row where all columns match. The scan does not see the rows that
the INSERT is about to INSERT, and DELETE stops with an error, "we could not
find the row to delete".

== Possible solutions ==

1. Implement gap locking in MyRocks. This will take some time.

2. Change parallel slave to wait *for commit*. This should only be done if
tables that are updated do not support Gap Locking. This is hard, it will
require making risky changes on the SQL layer.

3. Disallow parallel slave execution for tables without Gap Locking and without
primary key. (Looks most promising but I am not sure if it is sufficient).


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 56d339111a4: MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which makes optimizer always picks

[Sergey Petrunia]

Hi Varun,

On Mon, Jun 04, 2018 at 12:34:49PM +0530, Varun wrote:
> revision-id: 56d339111a4e5c868ffa3b1f2538c5f101f67767 
> (mariadb-10.0.30-366-g56d339111a4)
> parent(s): a61724a3ca187f6eb44fc67948acb76a94efa783
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-06-04 12:26:56 +0530
> message:
> 
> MDEV-16374: Filtered shows 0 for materilization scan for a semi join, which 
> makes optimizer always picks
> materialization scan over materialization lookup
> 
> For non-mergeable semi-joins we don't store the estimates of the IN subquery 
> in table->file->stats.records.
> In the function TABLE_LIST::fetch_number_of_rows, we store the number of rows 
> in the tables
> (estimates in case of derived table/views).
> Currently we don't store the estimates for non-mergeable semi-joins, which 
> leads to a problem of selecting
> materialization scan over materialization lookup.
> Fixed this by storing these estimated appropriately
> 

* This is a change in the optimizer. Should we really do it in 10.0? In my
opinion, this should go into 10.3 or 10.4.  Let's discuss this, and I think
Igor may have something to say, too.

* The patch updates selectivity.test but not selectivity_innodb.result (which
includes selectivity.test).


* Please see my last comments at https://jira.mariadb.org/browse/MDEV-16374 .
Here is an alternative patch (it passes the testsuite, and produces the same
result for selectivity* tes):

diff --git a/sql/sql_select.cc b/sql/sql_select.cc
index 1285835..884997c 100644
--- a/sql/sql_select.cc
+++ b/sql/sql_select.cc
@@ -6602,7 +6602,7 @@ double matching_candidates_in_table(JOIN_TAB *s, bool 
with_found_constraint,
   {
 TABLE *table= s->table;
 double sel= table->cond_selectivity;
-double table_records= (double)table->stat_records();
+double table_records= s->records; // psergey-10
 dbl_records= table_records * sel;
 return dbl_records;
   }

What do you think about it?
I would like to get Igor's opinion also because I'm not certain which member
variable should be changed here.

(Why does derived table modify table->stats.records while JTBM semi-joins
modify JOIN_TAB::records? These seem like two solutions for the same problem?)

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] unbundle RocksDB

[Sergey Petrunia]

Hi Michal,

On Wed, Jun 06, 2018 at 11:21:34AM +0200, Michal Schorm wrote:
> Since we have a rocksdb package ready in Fedora, I tried to unbundle it,
> using the system package instead.
> 
> I wasn't successful though.
> I haven't found any CMake argument which seemed related and from the build
> log it seems, the code tries to link against the static libraries in any
> case.
> 
> Before I'll start to dig further, I'm stopping by on this list, making sure
> it is possible.
>  - is it currently supported?
>  - have I overlooked any related CMake argument?

It is not currently supported.

MariaDB gets MyRocks (and RocksDB) from the upstream:
https://github.com/facebook/mysql-5.6/. Both upstream and MariaDB have RocksDB
as a submodule and so are tied to a particular revision of RocksDB.

Using a different revision of RocksDB will typically work, but 
- it is not a tested combination
- it will not necessarily work, especially if you use an older version of
RocksDB than the one that MyRocks used. RocksDB development continues, so 
there may be implicit dependencies that MyRocks at revision X requires 
RocksDB to be at least at revision Y.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] unbundle RocksDB

[Sergey Petrunia]

On Sat, Jun 09, 2018 at 05:49:15PM +0300, Sergey Petrunia wrote:
> Hi Michal,
> 
> On Wed, Jun 06, 2018 at 11:21:34AM +0200, Michal Schorm wrote:
> > Since we have a rocksdb package ready in Fedora, I tried to unbundle it,
> > using the system package instead.
> > 
> > I wasn't successful though.
> > I haven't found any CMake argument which seemed related and from the build
> > log it seems, the code tries to link against the static libraries in any
> > case.
> > 
> > Before I'll start to dig further, I'm stopping by on this list, making sure
> > it is possible.
> >  - is it currently supported?
> >  - have I overlooked any related CMake argument?
> 
> It is not currently supported.
> 
> MariaDB gets MyRocks (and RocksDB) from the upstream:
> https://github.com/facebook/mysql-5.6/. Both upstream and MariaDB have RocksDB
> as a submodule and so are tied to a particular revision of RocksDB.
> 
> Using a different revision of RocksDB will typically work, but 
> - it is not a tested combination
> - it will not necessarily work, especially if you use an older version of
> RocksDB than the one that MyRocks used. RocksDB development continues, so 
> there may be implicit dependencies that MyRocks at revision X requires 
> RocksDB to be at least at revision Y.
>
Also discussed this with the MyRocks upstream. Take-aways:
- A confirmation that unbundling is not a good idea.
- Sometimes, RocksDB features get deprecated/removed. So, there may also exist
implicit dependencies in form "MyRocks at revision X requires that RocksDB it
is using is *at most* revision Y".

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] MDEV-6111: Optimizer trace: on the first output

[Sergey Petrunia]

On Tue, Jun 12, 2018 at 07:40:08AM +0800, zhzhzoo zhang wrote:
> I've done
> condition processing and some range analysis.
> and for the test input (
> https://dev.mysql.com/doc/internals/en/tracing-example.html), It now
> outputs something like
> 

(not very readable from the email so I've made two pastes, from the current
tree and from MySQL 8):

https://gist.github.com/spetrunia/cb4a1a382a9e81a60491d524dbfc08f6
https://gist.github.com/spetrunia/62347cc3bf26e003909a32b378b6684f

> ```{
>   "join_optimization": {
> "select #": 1,
> "steps": {
>   "inner": {
> "select #": 1,

Two  'select #' :1 seem to be redundant.

There are also two nested "steps" elements which seem redundant as well.

> "steps": [
>   {
> "condition_processing": {
>   "condition": "WHERE",
>   "original_condition": "`test`.`alias1`.`pk` <> 0 and
> `test`.`alias2`.`pk` = `test`.`alias1`.`col_int_key`",
>   "steps": [
> {
>   "transformation": "equality_propagation",
>   "subselect_evaluation": [],
>   "resulting_condition": "`test`.`alias1`.`pk` <> 0 and
> multiple equal(`test`.`alias2`.`pk`, `test`.`alias1`.`col_int_key`)???"

What are the "???" at the end? Please get them removed.

also, if you check EXPLAIN FORMAT=JSON output, it prints conditions without
overquoting, like so:

"attached_condition": "alias1.pk <> 0 and alias1.col_int_key is not null"

This is done by using QT_EXPLAIN parameter:

  item->print(&str, QT_EXPLAIN);

Let's print them this way here as well.

> },
> {
>   "transformation": "constant_propagation",
>   "subselect_evaluation": [],
>   "resulting_condition": "`test`.`alias1`.`pk` <> 0 and
> multiple equal(`test`.`alias2`.`pk`, `test`.`alias1`.`col_int_key`)???"
> },

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] MDEV-6111: Optimizer trace: let's get rid of "optimizer_trace_enabled"

[Sergey Petrunia]

Hi Zhzhzoo,

I see that the current patch uses @@optimizer_trace_enabled variable to control
it.

Meanwhile, in MySQL the name of the variable is "optimizer_trace".

It is a flag set:

mysql>  select @@optimizer_trace;
+-+
| @@optimizer_trace   |
+-+
| enabled=on,one_line=off |
+-+
1 row in set (0.00 sec)

but one can set it to 1 or 0 to just set enabled on/off:

mysql> set @@optimizer_trace=1;
Query OK, 0 rows affected (0.00 sec)

mysql> set @@optimizer_trace=0;
Query OK, 0 rows affected (0.00 sec)

This obviously carries some legacy but I think we should have it the same way,
in order to be compatible with MySQL where possible.

Please use the same name/values as MySQL does.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] af15686: MDEV-16242: MyRocks: parallel slave on a table without PK can stop with ER_KEY_NOT_FOUND

[Sergey Petrunia]

Hi Kristian,

On Thu, Jul 05, 2018 at 05:11:44PM +0200, Kristian Nielsen wrote:
> pser...@askmonty.org (Sergei Petrunia) writes:
> 
> > MDEV-16242: MyRocks: parallel slave on a table without PK can stop with 
> > ER_KEY_NOT_FOUND
> >
> > DRAFT: If RBR event applier uses a secondary key or a full table scan
> > to locate a row, force waiting for prior commit to complete.
> 
> I think if you wait_for_prior_commit() also for secondary key case, you
> should definitely restrict it to be only for conservative parallel
> replication, not for optimistic/agressive. Using a secondary key seems
> generally fine (as opposed to full table scan), and this problem should
> affect only conservative mode, no need to penalise optimistic/aggressive.
> 
> (But maybe that is what is meant with "DRAFT").
>

Got it, thanks!

(This patch is indeed only for testing as Elena is still seeing some failures 
and I wanted to rule them out).

Btw, I also have figured that MyRocks wasn't making thd_rpl_deadlock_check()
calls and added these:
http://lists.askmonty.org/pipermail/commits/2018-June/012653.html
http://lists.askmonty.org/pipermail/commits/2018-June/012652.html
but I'm still in the process of trying to get the benchmark results for this.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] d277c4682e4: MDEV-16751: Server crashes in st_join_table::cleanup or TABLE_LIST::is_with_table_recursive_reference

[Sergey Petrunia]

On Sun, Jul 22, 2018 at 03:56:37PM +0530, Varun wrote:
> revision-id: d277c4682e4c6bb0e708296e6ae520c149dd78f4 
> (mariadb-5.5.60-47-gd277c4682e4)
> parent(s): 9cea4ccf12cb6e8746b9b440d9c62408a9ef04af
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-07-22 15:56:25 +0530
> message:
> 
> MDEV-16751: Server crashes in st_join_table::cleanup or 
> TABLE_LIST::is_with_table_recursive_reference
> with join_cache_level>2
> 
> During muliple equality propagation for a query in which we have an IN 
> subquery, the items in the select list of the
> subquery may not be part of the multiple equality because there might be 
> another occurence of the same field in the
> where clause of the subquery.
> So we keyuse_is_valid_for_access_in_chosen_plan function which expects the 
> items in the select list of the subquery to
> be same to the ones in the multiple equality (through these multiple 
> equalities we create keyuse array).
> The solution would be that we expect the same field not the same Item because 
> when we have SEMI JOIN MATERIALIZATION SCAN,
> we use copy back technique to copies back the materialised table fields to 
> the original fields of the base tables.
> 
> ---
>  mysql-test/r/subselect_mat.result| 35 +++
>  mysql-test/r/subselect_sj_mat.result | 35 +++
>  mysql-test/t/subselect_sj_mat.test   | 25 +
>  sql/sql_select.cc|  4 +++-
>  4 files changed, 98 insertions(+), 1 deletion(-)
...
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 6a64a0e9952..af6f63a55f1 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -7438,7 +7438,9 @@ bool 
> JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join,
>st_select_lex *sjm_sel= emb_sj_nest->sj_subq_pred->unit->first_select(); 
>for (uint i= 0; i < sjm_sel->item_list.elements; i++)
>{
> -if (sjm_sel->ref_pointer_array[i] == keyuse->val)
> +DBUG_ASSERT(keyuse->val->type() == Item::FIELD_ITEM);
^^(1)
> +Field *field = ((Item_field*)sjm_sel->ref_pointer_array[i])->field;
^^(2)
> +if (field->eq(((Item_field*)keyuse->val)->field))
>return true;
>}
>return false; 

As for typecast (2):
As far as I understand, it is valid because of the following:
- We get to this point in this function only when
  Semi-Join-Materialization-Scan is used.
- SJM-Scan is only used when the subquery's select list has Item_field objects
  (This is checked in subquery_types_allow_materialization()).

* Please add a comment in subquery_types_allow_materialization() about this
  function relying on that logic.
* Please add here an assert that  sjm_sel->ref_pointer_array[i] is an
  Item_field. The logic is complex and can be potentially broken.


As for typecast (1):
What if keyuse->val is not an Item_field?  If we get to this point in this 
function, it means it's an expression that only depends on the SJ-Inner table. 

This cannot just happen due to limitations in
subquery_types_allow_materialization(), and I wasn't able to construct an
example where it would happen (tried playing with multiple-equalities, but 
substitute_for_best_equal_item is called after get_best_combination(), 
tried playing with constant tables but then SJ-Materialization is not used,
etc. )

I think we will be safer by changing the assert to be this check:

  if (keyuse->val->type() == Item::FIELD_ITEM)
  {
... everything else
  }

The idea is that a KEYUSE that satisfies the requirement will still be
available, and we just want to filter out any extra KEYUSE that might pop up
due to some optimization.

what do you think?


Ok to push after the above is addressed.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 108af298032: MDEV-15454: Nested SELECT IN returns wrong results

[Sergey Petrunia]

On Fri, May 11, 2018 at 03:08:29AM +0530, Varun wrote:
> revision-id: 108af298032cb4cea316a08c304c306268a2badf 
> (mariadb-5.5.56-213-g108af298032)
> parent(s): 318097bb8f6e12c546b5dcd287416158209dbb39
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-05-11 02:56:33 +0530
> message:
> 
> MDEV-15454: Nested SELECT IN returns wrong results
> 
> In this case we are setting the field Item_func_eq::in_eqaulity_no for the 
> semi-join equalities.
> This helps us to remove these equalites as the inner tables are not available 
> during parent select execution
> while the outer tables are not available during materialization phase.
> We only have it set for the equalites for the fields involved with the IN 
> subquery
> and reset it for the equalities which do not belong to the IN subquery.
> 
> For example in case of nested IN subqueries:
> 
> SELECT t1.a FROM t1 WHERE t1.a IN
>   (SELECT t2.a FROM t2 where t2.b IN
>   (select t3.b from t3 where t3.c=27 ))
> 
> there are two equalites involving the fields of the IN subquery
> 
> 1) t2.b = t3.b :  the field Item_func_eq::in_eqaulity_no is set when we merge 
> the grandchild select into the child select
> 2) t1.a = t2.a :  the field Item_func_eq::in_eqaulity_no is set when we merge 
> the child select into the parent select
> 
> But when we perform case 2) we should ensure that we reset the equalities in 
> the child's WHERE clause.
> 
> ---
>  mysql-test/r/subselect_sj_mat.result | 46 +
>  mysql-test/t/subselect_sj_mat.test   | 49 ++
>  sql/opt_subselect.cc | 66 
> 
>  3 files changed, 161 insertions(+)
> 
...

> diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc
> index a7edd64e68b..31213a5cf1d 100644
> --- a/sql/opt_subselect.cc
> +++ b/sql/opt_subselect.cc
> @@ -1481,6 +1543,7 @@ static int subq_sj_candidate_cmp(Item_in_subselect* 
> el1, Item_in_subselect* el2,
>  TRUE   Out of memory error
>  */
>  
> +
>  static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect 
> *subq_pred)
>  {
>SELECT_LEX *parent_lex= parent_join->select_lex;
> @@ -1713,6 +1776,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, 
> Item_in_subselect *subq_pred)
>  
>if (subq_pred->left_expr->cols() == 1)
>{
> +reset_equality_number_for_subq_conds(sj_nest->sj_on_expr);
>  /* add left = select_list_element */
>  nested_join->sj_outer_expr_list.push_back(&subq_pred->left_expr);
>  /*
> @@ -1735,6 +1799,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, 
> Item_in_subselect *subq_pred)
>}
>else if (subq_pred->left_expr->type() == Item::ROW_ITEM)
>{
> +reset_equality_number_for_subq_conds(sj_nest->sj_on_expr);
>  /*
>disassemple left expression and add
>left1 = select_list_element1 and left2 = select_list_element2 ...
> @@ -1759,6 +1824,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, 
> Item_in_subselect *subq_pred)
>}
>else
>{
> +reset_equality_number_for_subq_conds(sj_nest->sj_on_expr);
>  /*
>add row operation
>left = (select_list_element1, select_list_element2, ...)

So, the code here is 

  if (...) 
  { 
variant1; 
  } 
  else if (...) 
  {
variant2;
  }
  else 
  {
variant3;
  }

and there is the same call added to all three:

> +reset_equality_number_for_subq_conds(sj_nest->sj_on_expr);

Can you move it out of the "if" and make it one call instead of three?

Otherwise, the patch is ok, ok to push after the above is addressed.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] f522fb9: MDEV-16934 Query with very large IN clause lists runs slowly

[Sergey Petrunia]

Hello Igor,

I am debugging with the patch and I see that PARAM::range_count is not reset
after the eq_ranges_exceeds_limit() call.

This causes different kinds of undesired effects. For example, here

  create table ten(a int);
  insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
  create table t1 (pk int, kp1 int, kp2 int, kp3 int, col1 int, 
key(kp1,kp2,kp3), primary key(pk));
  insert into t1 select A.a + 10*B.a+100* C.a, A.a, B.a , C.a, 1234 from ten A, 
ten B, ten C;
  alter table t1 add key3 int, add key2 int;
  update t1 set key3=kp1;
  update t1 set key2=kp1;
  alter table t1 add key(key2), add key(key3);
  set eq_range_index_dive_limit=10;
  explain format=json 
  select 1 from t1 force index (key2, key3) where key3=1 or key2=1;

index_merge will use Sort-Union instead of Union.


I assume we are targeting only IN (...) cases? Both our and MySQL's fixes will
still do many records_in_range calls for queries like 

  keypart1 IN (1,2,3,...) AND keypart2 IN (1,2,3,...) AND keypart3 >= 10

(because here the ranges will not be equality ranges). 

I assume this is outside of the scope of this fix? If so, then the patch is ok
to push after PARAM::range_count issue is resolved.

On Wed, Aug 15, 2018 at 07:48:30PM -0700, IgorBabaev wrote:
> revision-id: f522fb9f63f88195bccbbc0326a4a318cc2d3db0 
> (mariadb-10.2.16-50-gf522fb9)
> parent(s): 3c141e319ab29afdfe843553da385fe5d981906e
> author: Igor Babaev
> committer: Igor Babaev
> timestamp: 2018-08-15 19:48:30 -0700
> message:
> 
> MDEV-16934 Query with very large IN clause lists runs slowly
> 
> This patch introduces support for the system variable 
> eq_range_index_dive_limit
> that existed in MySQL starting from 5.6. The variable sets a limit for
> index dives into equality ranges. Index dives are performed by optimizer
> to estimate the number of rows in range scans. Index dives usually provide
> good estimate but they are pretty expensive. To estimate the number of rows
> in equality ranges statistical data on indexes can be employed. Its usage 
> gives
> not so good estimates but it's cheap. So if the number of equality dives
> required by an index scan exceeds the set limit no dives for equality
> ranges are performed by the optimizer for this index.
> 
> As the new system variable is introduced in a stable version the default
> value for it is set to a special value meaning there is no limit for the 
> number
> of index dives performed by the optimizer.
> 
> The patch partially uses the MySQL code for WL 5957
> 'Statistics-based Range optimization for many ranges'.
> 
> ---
>  mysql-test/r/mysqld--help.result   |  6 +++
>  mysql-test/r/range.result  | 41 +
>  mysql-test/r/range_mrr_icp.result  | 41 +
>  .../sys_vars/r/sysvars_server_embedded.result  | 14 ++
>  .../sys_vars/r/sysvars_server_notembedded.result   | 14 ++
>  mysql-test/t/range.test| 33 ++
>  sql/multi_range_read.cc| 11 +
>  sql/opt_range.cc   | 27 +++
>  sql/opt_range.h|  4 +-
>  sql/opt_range_mrr.cc   | 53 
> +++---
>  sql/sql_class.h|  1 +
>  sql/sql_statistics.h   |  2 +-
>  sql/sys_vars.cc| 10 
>  13 files changed, 238 insertions(+), 19 deletions(-)
> 
> diff --git a/mysql-test/r/mysqld--help.result 
> b/mysql-test/r/mysqld--help.result
> index 3b7d5e2..51ece33 100644
> --- a/mysql-test/r/mysqld--help.result
> +++ b/mysql-test/r/mysqld--help.result
> @@ -197,6 +197,11 @@ The following specify which files/extra groups are read 
> (specified before remain
>   cache, etc)
>   --enforce-storage-engine=name 
>   Force the use of a storage engine for new tables
> + --eq-range-index-dive-limit=# 
> + The optimizer will use existing index statistics instead
> + of doing index dives for equality ranges if the number of
> + equality ranges for the index is larger than or equal to
> + this number. If set to 0, index dives are always used.
>   --event-scheduler[=name] 
>   Enable the event scheduler. Possible values are ON, OFF,
>   and DISABLED (keep the event scheduler completely
> @@ -1259,6 +1264,7 @@ encrypt-binlog FALSE
>  encrypt-tmp-disk-tables FALSE
>  encrypt-tmp-files FALSE
>  enforce-storage-engine (No default value)
> +eq-range-index-dive-limit 0
>  event-scheduler OFF
>  expensive-subquery-limit 100
>  expire-logs-days 0
> diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result
> index 3a71d08..0c6be5e 100644
> --- a/mysql-test/r/range.result
> +++ b/mysql-test/r/range.result
> @@ -3002,5 +3002,46 @@ deallocate prepare stmt;
>  set optimizer_switch=@save_optimizer_switch;
>  drop table t1,t2,t3;
>  #
> +# MDEV-16934: using system variable eq_range_index_dive_limit
> +# 

Re: [Maria-developers] [Commits] 3c7e2540f63: MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with bad conversion

[Sergey Petrunia]

Hi Varun,

Looking at the testcase name, I have found Serg's patch for MDEV-4285. 

His patch skips sorting if the keys have zero length. He does it at a late
stage, in Filesort_buffer::sort_buffer().

His patch doesn't cover the example of MDEV-17020, because the testcase in
MDEV-17020 uses LIMIT and so uses a Priority Queue (and not
Filesort_buffer::sort_buffer which uses qsort).

However, your patch doesn't cover all cases either. There are other ways to
produce zero-length keys. I have found this one:

SELECT * FROM t1 ORDER BY 'foo', substring(pk, 0) LIMIT 2;

it crashes in the same way as the test for MDEV-17020 used to do.

Possible options:

1. Find all Items that might produce zero-length keys. It's difficult not to
   miss anything.

2. Instead of #1, add a universal logic "if sort key length is 0, don't sort".
   Is this doable?

3. Extend the Serg's approach. We cannot use his code directly ("dont sort"),
because Priority Queue API doesn't easily allow it, but perhaps we could add a
comparison function that will allow to compare zero-length keys? Or make them
1-byte long with a dummy value (do we need the sorting to be stable??)

On Fri, Aug 31, 2018 at 12:18:08AM +0530, Varun wrote:
> revision-id: 3c7e2540f63c6801a0d0a27fbea7c934bb61221f 
> (mariadb-10.0.36-13-g3c7e2540f63)
> parent(s): b3c320bb0b93e516cda4db277cfa3efeef48c988
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-08-31 00:14:20 +0530
> message:
> 
> MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY with 
> bad conversion
> 
> We have a case here where the the entire sort key is empty  and so sorting is 
> not needed.
> The crash happens because filesort expects the sort key to have length > 0.
> The case here is like ORDER BY CONVERT(a,char(0)), which if evaluated is a 
> constant so
> can be removed from the ORDER BY clause
> 
> ---
>  mysql-test/r/order_by_zerolength-4285.result | 28 
> +---
>  mysql-test/t/order_by_zerolength-4285.test   | 13 +
>  sql/item_timefunc.h  |  6 ++
>  3 files changed, 36 insertions(+), 11 deletions(-)
> 
> diff --git a/mysql-test/r/order_by_zerolength-4285.result 
> b/mysql-test/r/order_by_zerolength-4285.result
> index f60ce7d90c7..f77b284df3c 100644
> --- a/mysql-test/r/order_by_zerolength-4285.result
> +++ b/mysql-test/r/order_by_zerolength-4285.result
> @@ -12,15 +12,21 @@ pk
>  8
>  9
>  10
> -Warnings:
> -Warning  1292Truncated incorrect CHAR(0) value: '1'
> -Warning  1292Truncated incorrect CHAR(0) value: '2'
> -Warning  1292Truncated incorrect CHAR(0) value: '3'
> -Warning  1292Truncated incorrect CHAR(0) value: '4'
> -Warning  1292Truncated incorrect CHAR(0) value: '5'
> -Warning  1292Truncated incorrect CHAR(0) value: '6'
> -Warning  1292Truncated incorrect CHAR(0) value: '7'
> -Warning  1292Truncated incorrect CHAR(0) value: '8'
> -Warning  1292Truncated incorrect CHAR(0) value: '9'
> -Warning  1292Truncated incorrect CHAR(0) value: '10'
> +drop table t1;
> +#
> +# MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER BY 
> with bad conversion
> +#
> +set @save_sql_mode= @@sql_mode;
> +SET @@sql_mode= '';
> +CREATE TABLE t1 (pk INT PRIMARY KEY);
> +INSERT INTO t1 VALUES (1),(2);
> +explain
> +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  index   NULLPRIMARY 4   NULL2   Using 
> index
> +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2;
> +pk
> +1
> +2
> +set @@sql_mode= @save_sql_mode;
>  drop table t1;
> diff --git a/mysql-test/t/order_by_zerolength-4285.test 
> b/mysql-test/t/order_by_zerolength-4285.test
> index 2fb58edd36d..f03d528320c 100644
> --- a/mysql-test/t/order_by_zerolength-4285.test
> +++ b/mysql-test/t/order_by_zerolength-4285.test
> @@ -6,3 +6,16 @@ insert into t1 values 
> (1),(2),(3),(4),(5),(6),(7),(8),(9),(10);
>  select * from t1 order by now(), cast(pk as char(0));
>  drop table t1;
>  
> +--echo #
> +--echo # MDEV-17020: Assertion `length > 0' failed in ptr_compare upon ORDER 
> BY with bad conversion
> +--echo #
> +
> +set @save_sql_mode= @@sql_mode;
> +SET @@sql_mode= ''; 
> +CREATE TABLE t1 (pk INT PRIMARY KEY);
> +INSERT INTO t1 VALUES (1),(2);
> +explain
> +SELECT * FROM t1 ORDER BY 'foo', CONVERT(pk, CHAR(0)) LIMIT 2;
> +SELECT * FROM t1 ORDER BY 'foo', Cast(pk as CHAR(0)) LIMIT 2;
> +set @@sql_mode= @save_sql_mode;
> +drop table t1;
> diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h
> index 80b98758fb7..461c3feed8c 100644
> --- a/sql/item_timefunc.h
> +++ b/sql/item_timefunc.h
> @@ -903,6 +903,12 @@ class Item_char_typecast :public Item_str_func
>String *val_str(String *a);
>void fix_length_and_dec();
>void print(String *str, enum_query_type query_type);
> +  table_map used_tables() const

Re: [Maria-developers] [Commits] 1edec241473: MDEV-17137: Syntax errors with VIEW using MEDIAN

[Sergey Petrunia]

Hi Varun,

On Wed, Sep 12, 2018 at 06:13:23PM +0530, Varun wrote:
> revision-id: 1edec241473a122553fa9a27be995f3aef52654e 
> (mariadb-10.3.7-171-g1edec241473)
> parent(s): 5a1868b58d26b286b6ad433096e7184895953311
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-09-12 18:10:16 +0530
> message:
> 
> MDEV-17137: Syntax errors with VIEW using MEDIAN
> 
> The syntax error happened because we had not implemented a different print for
> percentile functions. The syntax is a bit different when we use percentile 
> functions
> as window functions in comparision to normal window functions.
> Implemented a seperate print function for percentile functions
> 
...
> diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h
> index b3e23748246..268182a2788 100644
> --- a/sql/item_windowfunc.h
> +++ b/sql/item_windowfunc.h
> @@ -1311,6 +1311,7 @@ class Item_window_func : public Item_func_or_sum
>bool resolve_window_name(THD *thd);
>
>void print(String *str, enum_query_type query_type);
> +  void print_percentile_functions(String *str, enum_query_type query_type);
>  
* Please make this function private
* I think "print_percentile_functions" is a bad name as it implies multiple
functions are printed. In fact, only one function is printed. How about
"print_for_percentile_functions"? 

Ok to push after the above is addressed.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] eq_range_index_dives_limit

[Sergey Petrunia]

Hello,

So, MDEV-16934 introduced eq_range_index_dives_limit into 10.2.8 and 10.3.0.

The default was set to 0 (which means no limit) in order to not introduce 
optimizer behavior change into stable versions.

The question is: should 10.4 also have 0 by default or we can set it to some
finite limit? MySQL's default value is 10.

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 737de39: Issue #885: Can't locate lib/mtr_process.pl in @INC on Ubuntu 18.04.1

[Sergey Petrunia]

Hi Kristian,

On Sat, Oct 27, 2018 at 07:25:29AM +0200, Kristian Nielsen wrote:
> Hi Sergey,
> 
> pser...@askmonty.org (Sergei Petrunia) writes:
> 
> > committer: Sergei Petrunia
> > branch nick: mysql-5.6-rocksdb-spetrunia
> 
> > Issue #885: Can't locate lib/mtr_process.pl in @INC on Ubuntu 18.04.1
> >
> > Add "." to @INC array like upstream does.
> 
> Just want to make sure you are aware of this:
> 
>   
> https://github.com/MariaDB/server/commit/d185f1d68bb1f37bea10d8ac6188e5a04faf4522
> 
> I see the branch nick is mysql-5.6-, so maybe this is like this
> for legacy reasons, which is fine. But this is basically the wrong fix.
> Clearly sometimes in ancient history, someone did not understand Perl
> `require` semantics, and ended up with a horrible mess. For example, if
> someone ever finds they need `require mtr_misc.pl` in a third file, they
> will find it will not work.
> 
> Anyway, just a pointer in case you were not aware of a small land-mine
> hidden under your feet ;-)
> 

Yes, I was aware of that patch.

The patch I've committed was for Facebook's MySQL branch, so I assumed copying 
MySQL's
solution would be more appropriate (I wasn't aware of its limitations though).

Thanks for the note,

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 0187bb03884: MDEV_-17589: Stack-buffer-overflow with indexed varchar (utf8) field

[Sergey Petrunia]

Hi Varun,

On Mon, Nov 05, 2018 at 07:36:17PM +0530, Varun wrote:
> revision-id: 0187bb03884a34ca8de96e47f4cb1f3d860e6db9 
> (mariadb-10.2.16-224-g0187bb03884)
> parent(s): 8a346f31b913daa011085afec2b2d38450c73e00
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-05 19:32:26 +0530
> message:
> 
> MDEV_-17589: Stack-buffer-overflow with indexed varchar (utf8) field
> 
> Create a new constant MAX_DATA_LENGTH_FOR_KEY.
> Replace the value of MAX_KEY_LENGTH to also include the LENGTH and NULL BYTES.
> 
...

> diff --git a/mysql-test/r/partition_datatype.result 
> b/mysql-test/r/partition_datatype.result
> index 2e518c194f0..66d003c97cb 100644
> --- a/mysql-test/r/partition_datatype.result
> +++ b/mysql-test/r/partition_datatype.result
> @@ -314,7 +314,7 @@ 
>  drop table t1;
>  set sql_mode='';
>  create table t1 (a varchar(3070)) partition by key (a);
> -ERROR HY000: The total length of the partitioning fields is too large
> +drop table t1;

I don't think it's a good idea that after this patch we allow the table
definitions that we didn't allow before. This raises the question of
datadir downgrades, etc.
I think we should only remove the buffer overrun.
Here's patch how to avoid the above:
https://gist.github.com/spetrunia/7e7e3fcc60a81a6cb854e77c12c367d3
>  create table t1 (a varchar(65532) not null) partition by key (a);
>  ERROR HY000: The total length of the partitioning fields is too large
>  create table t1 (a varchar(65533)) partition by key (a);
...

> diff --git a/mysql-test/r/innodb_ext_key.result 
> b/mysql-test/r/innodb_ext_key.result
> index c55e8d138f8..c76c8613c57 100644
> --- a/mysql-test/r/innodb_ext_key.result
> +++ b/mysql-test/r/innodb_ext_key.result
> @@ -1168,8 +1168,8 @@ EXPLAIN
>"access_type": "range",
>"possible_keys": ["f2"],
>"key": "f2",
> -  "key_length": "3070",
> -  "used_key_parts": ["f2", "pk1"],
> +  "key_length": "3074",
> +  "used_key_parts": ["f2", "pk1", "pk2"],

Same reaction as above:
This change shows that "Extended keys" feature now allows bigger set of key
extensions than before. Can we avoid making this change by changing
MAX_KEY_LENGTH to MAX_DATA_LENGTH_FOR_KEY somewhere?

>"rows": 1,
>"filtered": 100,
>"index_condition": "t1.pk1 <= 5 and t1.pk2 <= 5 and t1.f2 = 'abc'",

> diff --git a/sql/handler.h b/sql/handler.h
> index ed2ef822c88..75f72bdfb53 100644
> --- a/sql/handler.h
> +++ b/sql/handler.h
> @@ -378,6 +378,10 @@ enum enum_alter_inplace_result {
>  #define HA_KEY_NULL_LENGTH   1
>  #define HA_KEY_BLOB_LENGTH   2
>  
> +#define MAX_KEY_LENGTH (MAX_DATA_LENGTH_FOR_KEY \
> + +(MAX_REF_PARTS \
> +  *(HA_KEY_NULL_LENGTH + HA_KEY_BLOB_LENGTH)))
> +
I think this requires a comment. Please add something like

/* Maximum length of any index lookup key, in bytes */

>  #define HA_LEX_CREATE_TMP_TABLE  1U
>  #define HA_CREATE_TMP_ALTER 8U
>  
> @@ -3421,14 +3425,14 @@ class handler :public Sql_alloc
>uint max_key_parts() const
>{ return MY_MIN(MAX_REF_PARTS, max_supported_key_parts()); }
>uint max_key_length() const
> -  { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_length()); }
> +  { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_length()); }
>uint max_key_part_length() const
> -  { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_part_length()); }
> +  { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_part_length()); 
> }
>  
>virtual uint max_supported_record_length() const { return 
> HA_MAX_REC_LENGTH; }
>virtual uint max_supported_keys() const { return 0; }
>virtual uint max_supported_key_parts() const { return MAX_REF_PARTS; }
> -  virtual uint max_supported_key_length() const { return MAX_KEY_LENGTH; }
> +  virtual uint max_supported_key_length() const { return 
> MAX_DATA_LENGTH_FOR_KEY; }
>virtual uint max_supported_key_part_length() const { return 255; }
>virtual uint min_record_length(uint options) const { return 1; }
>  
> diff --git a/sql/sql_const.h b/sql/sql_const.h
> index 7395ae3c08a..0be63a2a5ad 100644
> --- a/sql/sql_const.h
> +++ b/sql/sql_const.h
> @@ -33,7 +33,7 @@
>  #define MAX_SYS_VAR_LENGTH 32
>  #define MAX_KEY MAX_INDEXES /* Max used keys */
>  #define MAX_REF_PARTS 32 /* Max parts used as ref */

Please add a comment, something like
/*
  Maximum length of the data part of an index lookup key. 
  
  The "data part" is defined as the value itself, not including the
  NULL-indicator bytes or varchar length bytes ("the Extras"). We need this 
  value because there was a bug where length of the Extras were not counted.

  You probably need MAX_KEY_LENGTH, not this constant.
*/

> -#define MAX_KEY_LENGTH 3072  /* max possible key */
> +#define MAX_DATA_LENGTH_FOR_KEY 3072
>  #if SIZEOF_OFF_T > 4
>  #define MAX_REFLENGTH 8  /* Max length for 
> record ref */
>  #else

Re: [Maria-developers] [Commits] e60cbbbda31: MDEV-12575: Server crash in AGGR_OP::put_record or in JOIN_CACHE::free or Invalid write in JOIN::make_aggr_tables_info

[Sergey Petrunia]

Hi Varun,

On Fri, Nov 09, 2018 at 02:26:04PM +0530, Varun wrote:
> revision-id: e60cbbbda31d7a10e87c16313171b2036a4519c4 
> (mariadb-10.2.18-52-ge60cbbbda31)
> parent(s): f8268f3cce4577c28ab62e53293556d05a74fb1a
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-01 14:27:24 +0530
> message:
> 
> MDEV-12575: Server crash in AGGR_OP::put_record or in JOIN_CACHE::free or 
> Invalid write in JOIN::make_aggr_tables_info
> 
> During the optimize state of a query, we come know that the result set
> would atmost contain one row, then for such a query we don't need
> to compute GROUP BY, ORDER BY and DISTINCT.
> 
...
> diff --git a/sql/sql_select.cc b/sql/sql_select.cc
> index 1fa80da85a6..a721aed6ce3 100644
> --- a/sql/sql_select.cc
> +++ b/sql/sql_select.cc
> @@ -2245,6 +2245,18 @@ JOIN::optimize_inner()
>if (!tables_list || !table_count)
>{
>  choose_tableless_subquery_plan();
> +
> +/* The output has atmost one row */
> +if (group_list)
> +{
> +  group_list= 0;
> +  group_optimized_away= 1;
> +  rollup.state= ROLLUP::STATE_NONE;
> +}
> +order=0;
> +simple_order=1;
> +select_distinct=0;
> +
Please fix the coding style:
- "x= y" (space after the =)
- use NULL for pointers
- use false for booleans

s/atmost/at most/

Ok to push after the above is addressed.

>  if (select_lex->have_window_funcs())
>  {
>if (!(join_tab= (JOIN_TAB*) thd->alloc(sizeof(JOIN_TAB
> ___
> commits mailing list
> comm...@mariadb.org
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] a9c1420233f: MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field

[Sergey Petrunia]

Hi Varun,

The patch is ok to push.

One last question: why is it made against 10.2? I see the MDEV issue has
AffectsVersion to start from 10.2, but to my knowledge the code and constants 
involved were not changed in 10.2, so the issue should be present in the
earlier versions as well?


On Mon, Nov 12, 2018 at 04:27:08PM +0530, Varun wrote:
> revision-id: a9c1420233f3f65aee00a8e7bdd1c8481b0bbc37 
> (mariadb-10.2.18-74-ga9c1420233f)
> parent(s): b290ef8c76e2d7dfbae7a85766694a6fd4648eac
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-12 16:25:25 +0530
> message:
> 
> MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field
> 
> Create a new constant MAX_DATA_LENGTH_FOR_KEY.
> Replace the value of MAX_KEY_LENGTH to also include the LENGTH and NULL BYTES.
> 
> ---
>  mysql-test/r/func_group_innodb.result | 23 +++
>  mysql-test/t/func_group_innodb.test   | 18 ++
>  sql/handler.h | 12 +---
>  sql/partition_info.cc |  4 ++--
>  sql/sql_const.h   | 13 -
>  sql/table.cc  |  2 +-
>  6 files changed, 65 insertions(+), 7 deletions(-)
> 
> diff --git a/mysql-test/r/func_group_innodb.result 
> b/mysql-test/r/func_group_innodb.result
> index 52d5922df95..c2bfe9bf81c 100644
> --- a/mysql-test/r/func_group_innodb.result
> +++ b/mysql-test/r/func_group_innodb.result
> @@ -246,4 +246,27 @@ EXPLAIN SELECT MIN(c) FROM t1 GROUP BY b;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1SIMPLE  t1  range   NULLb   263 NULL3   Using 
> index for group-by
>  DROP TABLE t1;
> +#
> +# MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field
> +#
> +CREATE TABLE t1 (v1 varchar(1020), v2 varchar(2), v3 varchar(2), KEY k1 
> (v3,v2,v1)) ENGINE=InnoDB CHARACTER SET=utf8;
> +INSERT INTO t1 VALUES ('king', 'qu','qu'), ('bad','go','go');
> +explain
> +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu';
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  NULLNULLNULLNULLNULLNULLNULLSelect 
> tables optimized away
> +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu';
> +MIN(t1.v1)
> +king
> +drop table t1;
> +CREATE TABLE t1 (v1 varchar(1024) CHARACTER SET utf8, KEY v1 (v1)) 
> ENGINE=InnoDB;
> +INSERT INTO t1 VALUES ('king'), ('bad');
> +explain
> +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  NULLNULLNULLNULLNULLNULLNULLNo 
> matching min/max row
> +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x;
> +MIN(x.v1)
> +NULL
> +drop table t1;
>  End of 5.5 tests
> diff --git a/mysql-test/t/func_group_innodb.test 
> b/mysql-test/t/func_group_innodb.test
> index c62d3d08496..0d24f37363b 100644
> --- a/mysql-test/t/func_group_innodb.test
> +++ b/mysql-test/t/func_group_innodb.test
> @@ -192,4 +192,22 @@ EXPLAIN SELECT MIN(c) FROM t1 GROUP BY b;
>  
>  DROP TABLE t1;
>  
> +--echo #
> +--echo # MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field
> +--echo #
> +
> +CREATE TABLE t1 (v1 varchar(1020), v2 varchar(2), v3 varchar(2), KEY k1 
> (v3,v2,v1)) ENGINE=InnoDB CHARACTER SET=utf8;
> +INSERT INTO t1 VALUES ('king', 'qu','qu'), ('bad','go','go');
> +explain
> +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu';
> +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu';
> +drop table t1;
> +
> +CREATE TABLE t1 (v1 varchar(1024) CHARACTER SET utf8, KEY v1 (v1)) 
> ENGINE=InnoDB;
> +INSERT INTO t1 VALUES ('king'), ('bad');
> +explain
> +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x;
> +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x;
> +drop table t1;
> +
>  --echo End of 5.5 tests
> diff --git a/sql/handler.h b/sql/handler.h
> index ed2ef822c88..6745312a17d 100644
> --- a/sql/handler.h
> +++ b/sql/handler.h
> @@ -378,6 +378,12 @@ enum enum_alter_inplace_result {
>  #define HA_KEY_NULL_LENGTH   1
>  #define HA_KEY_BLOB_LENGTH   2
>  
> +/* Maximum length of any index lookup key, in bytes */
> +
> +#define MAX_KEY_LENGTH (MAX_DATA_LENGTH_FOR_KEY \
> + +(MAX_REF_PARTS \
> +  *(HA_KEY_NULL_LENGTH + HA_KEY_BLOB_LENGTH)))
> +
>  #define HA_LEX_CREATE_TMP_TABLE  1U
>  #define HA_CREATE_TMP_ALTER 8U
>  
> @@ -3421,14 +3427,14 @@ class handler :public Sql_alloc
>uint max_key_parts() const
>{ return MY_MIN(MAX_REF_PARTS, max_supported_key_parts()); }
>uint max_key_length() const
> -  { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_length()); }
> +  { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_length()); }
>uint max_key_part_length() const
> -  { return MY_MIN(MAX_KEY_LENGTH, max_supp

Re: [Maria-developers] [Commits] 3f4dab80912: MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY

[Sergey Petrunia]

Hi Varun,

On Mon, Nov 12, 2018 at 02:41:51PM +0530, Varun wrote:
> revision-id: 3f4dab80912682325b61b288178febe9ee55de49 
> (mariadb-10.0.36-78-g3f4dab80912)
> parent(s): 6cecb10a2f8b6536bed78ab6d3791d8befc9d732
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-12 14:40:07 +0530
> message:
> 
> MDEV-17032: Estimates are higher for partitions of a table with 
> @@use_stat_tables= PREFERABLY
> 
> The problem here is EITS statistics does not calculate statistics for the 
> partitions of the table.
> So a temporary solution would be to not read EITS statistics for partitioned 
> tables.
> 
> Also disabling reading of EITS for columns that participate in the partition 
> list of a table.
> 
> ---
>  mysql-test/r/partition.result | 37 +
>  mysql-test/t/partition.test   | 35 +++
>  sql/opt_range.cc  | 12 ++--
>  sql/partition_info.cc | 21 +
>  sql/partition_info.h  |  1 +
>  sql/sql_statistics.cc |  6 ++
>  6 files changed, 110 insertions(+), 2 deletions(-)
> 
> diff --git a/mysql-test/r/partition.result b/mysql-test/r/partition.result
> index c6669176b3d..abfbc21dccb 100644
> --- a/mysql-test/r/partition.result
> +++ b/mysql-test/r/partition.result
> @@ -2645,3 +2645,40 @@ Warnings:
>  Note 1517Duplicate partition name p2
>  DEALLOCATE PREPARE stmt;
>  DROP TABLE t1;
> +#
> +# MDEV-17032: Estimates are higher for partitions of a table with 
> @@use_stat_tables= PREFERABLY
> +#
> +create table ten(a int);
> +insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
> +create table one_k(a int);
> +insert into one_k select A.a + B.a* 10 + C.a * 100 from ten A, ten B, ten C;

Please rename the tables to follow the t0,t1,t2... convention.
> +create table t1 (
> +part_key int,
> +a int,
> +b int
> +) partition by list(part_key) (
> +partition p0 values in (0),
> +partition p1 values in (1),
> +partition p2 values in (2),
> +partition p3 values in (3),
> +partition p4 values in (4)
> +);
> +insert into t1
> +select mod(a,5), a/100, a from one_k;
> +set @save_use_stat_tables= @@use_stat_tables;
> +set 
> @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity;
> +set @@use_stat_tables= PREFERABLY;
> +set @@optimizer_use_condition_selectivity=4;
> +analyze table t1;
> +TableOp  Msg_typeMsg_text
> +test.t1  analyze status  Engine-independent statistics collected
> +test.t1  analyze status  OK

After some time passes, it gets hard to understand what exactly is being
checked. Then, there's some change in the EXPLAIN (typically for an unrelated
reason) and one is left wondering, whether the new EXPLAIN output is ok or not.

* please use EXPLAIN PARTITIONS.
* please add something like "--echo # rows should be X, not Y" before the
query.

> +explain select * from t1 where  part_key in (1,2);
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  ALL NULLNULLNULLNULL400 Using 
> where
> +explain select * from t1 where part_key > 1;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
> +1SIMPLE  t1  ALL NULLNULLNULLNULL600 Using 
> where

This does not provide sufficient coverage. I get this result if I just apply
the change in sql_statistics.cc.

mysql> explain select * from t1 where  part_key in (1,2);
+--+-+---+--+---+--+-+--+--+-+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | 
rows | Extra   |
+--+-+---+--+---+--+-+--+--+-+
|1 | SIMPLE  | t1| ALL  | NULL  | NULL | NULL| NULL |  
400 | Using where |
+--+-+---+--+---+--+-+--+--+-+
1 row in set (0.01 sec)

mysql> explain select * from t1 where part_key > 1;
+--+-+---+--+---+--+-+--+--+-+
| id   | select_type | table | type | possible_keys | key  | key_len | ref  | 
rows | Extra   |
+--+-+---+--+---+--+-+--+--+-+
|1 | SIMPLE  | t1| ALL  | NULL  | NULL | NULL| NULL |  
600 | Using where |
+--+-+---+--+---+--+-+--+--+-+
1 row in set (0.00 sec)

Please add testcases for other changes.

> +set 
> @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity;
> +set @@use_stat_tables= @save_use_stat_tables;
> +drop table t1, one_k, ten;
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index 3bcaa72e32f..993b7d57e0a 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -3322,6 +3322,10 @@ bo

Re: [Maria-developers] [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and ANALYZE TABLE

[Sergey Petrunia]

Hi Varun,

On Fri, Nov 16, 2018 at 11:48:59PM +0530, Varun wrote:
> revision-id: 68b7e231cb3a1cdfbd968c2cbf72687b29b6d2da 
> (mariadb-10.3.6-209-g68b7e231cb3)
> parent(s): b9a9055793ab8b9a50200e2f55602463627dedd3

Does the above say the patch is going into 10.3? 
The MDEV has fixVersion=10.4 and the parent revision is also in 10.4.
So I assume the target version is 10.4? 

> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-16 23:48:11 +0530
> message:
> 
> MDEV-17255: New optimizer defaults and ANALYZE TABLE
> 
> Added another value for the use_stat_tables system variable

Please mention in the commit comment:
- the name of the new value
- the fact that the new value is now the default.

> ---
>  sql/opt_range.cc | 2 +-
>  sql/sql_admin.cc | 3 ++-
>  sql/sql_statistics.h | 1 +
>  sql/sys_vars.cc  | 4 ++--
>  4 files changed, 6 insertions(+), 4 deletions(-)

Please run the testsuite. I did the 'main' testsuite and:
- main.mysqld--help, main.stat_tables_disabled - these need updates
- a few other tests need to be updated as now opening any table will now cause
  the server to attept to read EITS stats for it.

Please also add test coverage for this particular MDEV. This should be a
testcase that shows that 

- with PREFERABLY, ANALYZE will collect EITS stats, SELECT will use them.

- with PREFERABLY_FOR_READS :
--  ANALYZE will not collect EITS stats
--  ANALYZE .. PERSISTENT FOR ... will update  the EITS stats.
-- SELECT will use them.

.
> 
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index d6db365a8a2..f501a5ae085 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -3158,7 +3158,7 @@ bool calculate_cond_selectivity_for_table(THD *thd, 
> TABLE *table, Item **cond)
>  
>if (thd->variables.optimizer_use_condition_selectivity > 2 &&
>!bitmap_is_clear_all(used_fields) &&
> -  thd->variables.use_stat_tables > 0)
> +  get_use_stat_tables_mode(thd) > NEVER)
>{
>  PARAM param;
>  MEM_ROOT alloc;
> diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc
> index d0d959de8f9..b5c732737b7 100644
> --- a/sql/sql_admin.cc
> +++ b/sql/sql_admin.cc
> @@ -767,7 +767,8 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* 
> tables,
>}
>collect_eis=
>  (table->table->s->table_category == TABLE_CATEGORY_USER &&
> - (get_use_stat_tables_mode(thd) > NEVER ||
> + ((get_use_stat_tables_mode(thd) > NEVER && 
> +get_use_stat_tables_mode(thd) < PREFERABLY_FOR_READS)  ||
>lex->with_persistent_for_clause));
>  
>  
> diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h
> index 39cddf95188..a942c05be09 100644
> --- a/sql/sql_statistics.h
> +++ b/sql/sql_statistics.h
> @@ -22,6 +22,7 @@ enum enum_use_stat_tables_mode
>NEVER,
>COMPLEMENTARY,
>PREFERABLY,
Please add a comment describing the new value.
> +  PREFERABLY_FOR_READS,
>  } Use_stat_tables_mode;
>  
>  typedef
> diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc
> index 92c7d329bb9..3652d728fc1 100644
> --- a/sql/sys_vars.cc
> +++ b/sql/sys_vars.cc
> @@ -5841,12 +5841,12 @@ static Sys_var_ulong Sys_progress_report_time(
> VALID_RANGE(0, UINT_MAX), DEFAULT(5), BLOCK_SIZE(1));
>  
>  const char *use_stat_tables_modes[] =
> -   {"NEVER", "COMPLEMENTARY", "PREFERABLY", 0};
> +   {"NEVER", "COMPLEMENTARY", "PREFERABLY", "PREFERABLY_FOR_READS", 
> 0};
>  static Sys_var_enum Sys_optimizer_use_stat_tables(
> "use_stat_tables",
> "Specifies how to use system statistics tables",
> SESSION_VAR(use_stat_tables), CMD_LINE(REQUIRED_ARG),
> -   use_stat_tables_modes, DEFAULT(0));
> +   use_stat_tables_modes, DEFAULT(3));
>  
>  static Sys_var_ulong Sys_histogram_size(
> "histogram_size",

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 1ed2d8b98ad: MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY

[Sergey Petrunia]

Hi Varun,

On Fri, Nov 16, 2018 at 07:41:07PM +0530, Varun wrote:
> revision-id: 1ed2d8b98ade099fe23b7d5c00d23364388e15aa 
> (mariadb-10.0.36-80-g1ed2d8b98ad)
> parent(s): a84d87fde8c0bc325c8e00f06ea02bcd84a75d55
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-16 19:40:47 +0530
> message:
> 
> MDEV-17032: Estimates are higher for partitions of a table with 
> @@use_stat_tables= PREFERABLY
> 
> The problem here is EITS statistics does not calculate statistics for the 
> partitions of the table.
> So a temporary solution would be to not read EITS statistics for partitioned 
> tables.
> 
> Also disabling reading of EITS for columns that participate in the partition 
> list of a table.
> 
> ---
>  mysql-test/r/partition.result | 88 
> +++
>  mysql-test/t/partition.test   | 58 
>  sql/opt_range.cc  | 12 +-
>  sql/partition_info.cc | 26 +
>  sql/partition_info.h  |  1 +
>  sql/sql_statistics.cc | 16 
>  6 files changed, 199 insertions(+), 2 deletions(-)
> 
> diff --git a/mysql-test/r/partition.result b/mysql-test/r/partition.result
> index c6669176b3d..aedf9f89f0e 100644
> --- a/mysql-test/r/partition.result
> +++ b/mysql-test/r/partition.result
> @@ -2645,3 +2645,91 @@ Warnings:
>  Note 1517Duplicate partition name p2
>  DEALLOCATE PREPARE stmt;
>  DROP TABLE t1;
> +#
> +# MDEV-17032: Estimates are higher for partitions of a table with 
> @@use_stat_tables= PREFERABLY
> +#
> +create table t0(a int);
> +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
> +create table t1(a int);
> +insert into t1 select A.a + B.a* 10 + C.a * 100 from t0 A, t0 B, t0 C;
> +create table t2 (
> +part_key int,
> +a int,
> +b int
> +) partition by list(part_key) (
> +partition p0 values in (0),
> +partition p1 values in (1),
> +partition p2 values in (2),
> +partition p3 values in (3),
> +partition p4 values in (4)
> +);
> +insert into t2
> +select mod(a,5), a/100, a from t1;
> +set @save_use_stat_tables= @@use_stat_tables;
> +set 
> @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity;
> +#
> +# Tests using stats provided by the storage engine
> +#
> +explain extended select * from t2 where part_key=1;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsfilteredExtra
> +1SIMPLE  t2  ALL NULLNULLNULLNULL200 100.00  
> Using where
> +Warnings:
> +Note 1003select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS 
> `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` = 1)
> +explain partitions select * from t2 where part_key=1;
> +id   select_type table   partitions  typepossible_keys   key 
> key_len ref rowsExtra
> +1SIMPLE  t2  p1  ALL NULLNULLNULLNULL200 
> Using where
> +explain extended select * from t2 where  part_key in (1,2);
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsfilteredExtra
> +1SIMPLE  t2  ALL NULLNULLNULLNULL400 100.00  
> Using where
> +Warnings:
> +Note 1003select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS 
> `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` in 
> (1,2))
> +explain partitions select * from t2 where part_key=1;
> +id   select_type table   partitions  typepossible_keys   key 
> key_len ref rowsExtra
> +1SIMPLE  t2  p1  ALL NULLNULLNULLNULL200 
> Using where
> +explain extended select * from t2 where  part_key in (1,2);
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsfilteredExtra
> +1SIMPLE  t2  ALL NULLNULLNULLNULL400 100.00  
> Using where
> +Warnings:
> +Note 1003select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS 
> `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` in 
> (1,2))
> +explain partitions select * from t2 where  part_key in (1,2);
> +id   select_type table   partitions  typepossible_keys   key 
> key_len ref rowsExtra
> +1SIMPLE  t2  p1,p2   ALL NULLNULLNULLNULL400 
> Using where
> +set @save_histogram_size=@@histogram_size;
> +set @@histogram_size=100;
> +set @@use_stat_tables= PREFERABLY;
> +set @@optimizer_use_condition_selectivity=4;
> +analyze table t2;
> +TableOp  Msg_typeMsg_text
> +test.t2  analyze status  Engine-independent statistics collected
> +test.t2  analyze status  OK
> +#
> +# Tests using EITS
> +#
> +explain extended select * from t2 where part_key=1;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsfilteredExtra
> +1SIMPLE  t2  ALL NULLNULLNULLNULL20

[Maria-developers] Fwd: Re: [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and ANALYZE TABLE

[Sergey Petrunia]

- Forwarded message from Igor Babaev  -

Date: Sun, 25 Nov 2018 11:45:34 -0800
From: Igor Babaev 
To: Sergey Petrunia 
Subject: Fwd: Re: [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and 
ANALYZE TABLE

Hi Sergey,

PREFERABLY_FOR_READS

sounds confusing, because the stat tables still are used for UPDATE/DELETE.

Maybe

PREFERABLY_FOR_QUERIES?

Besides, don't forget that you need

COMPLEMENTARY_FOR_READS[QUERIES]

as well.

Regards,

Igor.



 Forwarded Message 
Subject:Re: [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults 
and
ANALYZE TABLE
Date:   Sat, 24 Nov 2018 21:47:57 +0300
From:   Sergey Petrunia 
Reply-To:   maria-developers@lists.launchpad.net
To: Varun , maria-developers@lists.launchpad.net
CC: comm...@mariadb.org



Hi Varun,

On Fri, Nov 16, 2018 at 11:48:59PM +0530, Varun wrote:
> revision-id: 68b7e231cb3a1cdfbd968c2cbf72687b29b6d2da 
> (mariadb-10.3.6-209-g68b7e231cb3)
> parent(s): b9a9055793ab8b9a50200e2f55602463627dedd3

Does the above say the patch is going into 10.3?
The MDEV has fixVersion=10.4 and the parent revision is also in 10.4.
So I assume the target version is 10.4?

> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-11-16 23:48:11 +0530
> message:
> 
> MDEV-17255: New optimizer defaults and ANALYZE TABLE
> 
> Added another value for the use_stat_tables system variable

Please mention in the commit comment:
- the name of the new value
- the fact that the new value is now the default.

> ---
>  sql/opt_range.cc | 2 +-
>  sql/sql_admin.cc | 3 ++-
>  sql/sql_statistics.h | 1 +
>  sql/sys_vars.cc  | 4 ++--
>  4 files changed, 6 insertions(+), 4 deletions(-)

Please run the testsuite. I did the 'main' testsuite and:
- main.mysqld--help, main.stat_tables_disabled - these need updates
- a few other tests need to be updated as now opening any table will now cause
  the server to attept to read EITS stats for it.

Please also add test coverage for this particular MDEV. This should be a
testcase that shows that

- with PREFERABLY, ANALYZE will collect EITS stats, SELECT will use them.

- with PREFERABLY_FOR_READS :
--  ANALYZE will not collect EITS stats
--  ANALYZE .. PERSISTENT FOR ... will update  the EITS stats.
-- SELECT will use them.

.
> 
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index d6db365a8a2..f501a5ae085 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -3158,7 +3158,7 @@ bool calculate_cond_selectivity_for_table(THD *thd, 
> TABLE *table, Item **cond)
> 
>if (thd->variables.optimizer_use_condition_selectivity > 2 &&
>!bitmap_is_clear_all(used_fields) &&
> -  thd->variables.use_stat_tables > 0)
> +  get_use_stat_tables_mode(thd) > NEVER)
>{
>  PARAM param;
>  MEM_ROOT alloc;
> diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc
> index d0d959de8f9..b5c732737b7 100644
> --- a/sql/sql_admin.cc
> +++ b/sql/sql_admin.cc
> @@ -767,7 +767,8 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* 
> tables,
>}
>collect_eis=
>  (table->table->s->table_category == TABLE_CATEGORY_USER &&
> - (get_use_stat_tables_mode(thd) > NEVER ||
> + ((get_use_stat_tables_mode(thd) > NEVER &&
> +get_use_stat_tables_mode(thd) < PREFERABLY_FOR_READS)  ||
>lex->with_persistent_for_clause));
> 
> 
> diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h
> index 39cddf95188..a942c05be09 100644
> --- a/sql/sql_statistics.h
> +++ b/sql/sql_statistics.h
> @@ -22,6 +22,7 @@ enum enum_use_stat_tables_mode
>NEVER,
>COMPLEMENTARY,
>PREFERABLY,
Please add a comment describing the new value.
> +  PREFERABLY_FOR_READS,
>  } Use_stat_tables_mode;
> 
>  typedef
> diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc
> index 92c7d329bb9..3652d728fc1 100644
> --- a/sql/sys_vars.cc
> +++ b/sql/sys_vars.cc
> @@ -5841,12 +5841,12 @@ static Sys_var_ulong Sys_progress_report_time(
> VALID_RANGE(0, UINT_MAX), DEFAULT(5), BLOCK_SIZE(1));
> 
>  const char *use_stat_tables_modes[] =
> -   {"NEVER", "COMPLEMENTARY", "PREFERABLY", 0};
> +   {"NEVER", "COMPLEMENTARY", "PREFERABLY", "PREFERABLY_FOR_READS", 
> 0};
>  static Sys_var_enum Sys_optimizer_use_stat_tables(
> "use_stat_tables",
> "Specifies how to use system statistics tables",
> SESSION_VAR(use_stat_tables), CMD_LINE(REQUIRED_ARG),
> -   use_stat_tables_modes, DEFAULT(0));
> +   use_stat_tables_modes, DEFAULT(3));
> 
>  static Sys_var_ulong Sys_histogram_size(
> "histogram_size&quo

[Maria-developers] Do we support compilers that support C++11 but not thread_local?

[Sergey Petrunia]

Hi,

I'm writing this in connection with MyRocks Storage Engine and
https://github.com/facebook/mysql-5.6/issues/904

C++ 11 includes "thread_local" specifier : 
https://en.cppreference.com/w/cpp/keyword/thread_local

GNU also has __thread, which was available before C++11.

RocksDB (and so MyRocks) uses both __thread and thread_local. thread_local is
part of the standard, so they might switch to thread_local.
The question: will it cause any issues for us?

In other words, do we have compilers/environments that 
- claim to support C++11 (so that we attempt to compile MyRocks)
- but do not support "thread_local" 
?

For Windows, RocksDB does this in port/win/port_win.h:

#ifndef __thread
#define __thread __declspec(thread)
#endif

but this page page about Visual Studio 2015)  
https://msdn.microsoft.com/en-us/library/6yh4a9k1.aspx

says that

  C++11: The thread_local storage class specifier is the recommended 
  way to specify thread-local storage for objects and class members.

I assume this means it's ok to use thread_local on Windows?

RocksDB source code mentions one problematic case:

  // However, thread_local is not supported in all compilers that accept 
-std=c++11
  // (e.g., eg Mac with XCode < 8. XCode 8+ supports thread_local).

As far as I understand this is not relevant for us?

Anyone aware of other such cases?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] MDEV-16188: Rowid filtering: EXPLAIN FORMAT=JSON support

[Sergey Petrunia]

Hello Igor,

I have pushed into 10.4-mdev16188 tree a patch that adds EXPLAIN FORMAT=JSON
support. There is no ANALYZE support, yet.

main.rowid_filter has an example of EXPLAIN FORMAT=JSON at the bottom. The
filter is described like so:

+  "rowid_filter": {
+"range": {
+  "key": "i_l_shipdate",
+  "used_key_parts": ["l_shipDATE"]
+},
+"rows": 8,
+"selectivity_pct": 0.1332
+  },

Here "rows" is how many rows we will get from the quick select that is used to
build the filter.

selectivity_pct is the selectivity of the filter.

You will also notice this change from (1%) to (0%):

-1  SIMPLE  lineitemrange|filter
PRIMARY,i_l_shipdate,i_l_receiptdate,i_l_orderkey,i_l_orderkey_quantity 
i_l_receiptdate|i_l_shipdate4|4 NULL  6 (1%)  Using index 
condition; Using where; Using filter
+1  SIMPLE  lineitemrange|filter
PRIMARY,i_l_shipdate,i_l_receiptdate,i_l_orderkey,i_l_orderkey_quantity 
i_l_receiptdate|i_l_shipdate4|4 NULL  6 (0%)  Using index 
condition; Using where; Using filter

This is because the original code had a piece of logic in
JOIN_TAB::save_filter_explain_data: print "1%" if it was "0%".

I'm wondering, is there any reason we cannot print the number with greater
precision in this case?


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 2da93752430: MDEV-17255: New optimizer defaults and ANALYZE TABLE

[Sergey Petrunia]

Hi Varun,

Nice to see that most of the input has been addressed.

However, the issue of missing test coverage is still not resolved:

On Sat, Nov 24, 2018 at 09:47:57PM +0300, Sergey Petrunia wrote:
>> Please also add test coverage for this particular MDEV. This should be a
>> testcase that shows that 
>> 
>> - with PREFERABLY, ANALYZE will collect EITS stats, SELECT will use them.
>> 
>> - with PREFERABLY_FOR_READS :
>> --  ANALYZE will not collect EITS stats
>> --  ANALYZE .. PERSISTENT FOR ... will update  the EITS stats.
>> -- SELECT will use them.

and now it is still not addressed. I made this change:

@@ -104,6 +104,8 @@ Use_stat_tables_mode get_use_stat_tables_mode(THD *thd)
 inline
 bool check_eits_collection_allowed(THD *thd)
 {
+  if (get_use_stat_tables_mode(thd) == PREFERABLY_FOR_QUERIES)
+DBUG_ASSERT(0);
   return (get_use_stat_tables_mode(thd) == COMPLEMENTARY ||
   get_use_stat_tables_mode(thd) == PREFERABLY);
 }

and the testsuite still passes!


On Wed, Dec 05, 2018 at 07:57:06PM +0530, Varun wrote:
> revision-id: 2da93752430d0dd07344c9d62f2468bba8962b77 
> (mariadb-10.3.6-266-g2da93752430)
> parent(s): e0739064450f2c2be6d5de1d799582121747dd39
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-12-05 19:56:23 +0530
> message:
> 
> MDEV-17255: New optimizer defaults and ANALYZE TABLE
> 
> Added to new values to the server variable use_stat_tables.
> The values are COMPLEMENTARY_FOR_QUERIES and PREFERABLY_FOR_QUERIES.
> Both these values don't allow to collect EITS for queries like
> analyze table t1;
> To collect EITS we would need to use the syntax with persistent like
>analyze table t1 persistent for columns (col1,col2...) index (idx1, 
> idx2...) / ALL
> 
> Changing the default value from NEVER to PREFERABLY_FOR_QUERIES.
> 
> ---
>  mysql-test/include/default_mysqld.cnf |  1 +
>  mysql-test/main/stat_tables.result| 59 
> +++
>  mysql-test/main/stat_tables.test  | 43 ++
>  mysql-test/main/stat_tables_innodb.result | 59 
> +++
>  sql/sql_admin.cc  |  2 +-
>  sql/sql_statistics.cc |  5 ++-
>  sql/sql_statistics.h  | 27 ++
>  sql/sys_vars.cc   |  5 +--
>  8 files changed, 195 insertions(+), 6 deletions(-)
> 
> diff --git a/mysql-test/include/default_mysqld.cnf 
> b/mysql-test/include/default_mysqld.cnf
> index 69a2b58288b..5ba3bdeb92c 100644
> --- a/mysql-test/include/default_mysqld.cnf
> +++ b/mysql-test/include/default_mysqld.cnf
> @@ -107,6 +107,7 @@ 
> loose-performance-schema-consumer-thread-instrumentation=ON
>  binlog-direct-non-transactional-updates
>  
>  default-storage-engine=myisam
> +use-stat-tables=preferably_for_queries
>  
>  loose-ssl-ca=@ENV.MYSQL_TEST_DIR/std_data/cacert.pem
>  loose-ssl-cert=@ENV.MYSQL_TEST_DIR/std_data/server-cert.pem
> diff --git a/mysql-test/main/stat_tables.result 
> b/mysql-test/main/stat_tables.result
> index 308529ece47..67bf3f9237d 100644
> --- a/mysql-test/main/stat_tables.result
> +++ b/mysql-test/main/stat_tables.result
> @@ -605,4 +605,63 @@ SELECT MAX(pk) FROM t1;
>  MAX(pk)
>  NULL
>  DROP TABLE t1;
> +#
> +# MDEV-17255: New optimizer defaults and ANALYZE TABLE
> +#
> +set @save_use_stat_tables= @@use_stat_tables;
> +create table t1 (a int, b int);
> +insert into t1(a,b) values 
> (1,2),(1,3),(1,4),(1,5),(2,6),(2,7),(3,8),(3,9),(3,9),(4,10);
> +#
> +# with use_stat_tables= PREFERABLY_FOR_QUERIES
> +# analyze table t1 will not collect statistics
> +#
> +analyze table t1;
> +TableOp  Msg_typeMsg_text
> +test.t1  analyze status  Engine-independent statistics collected
> +test.t1  analyze status  OK
> +select * from mysql.column_stats;
> +db_name  table_name  column_name min_value   max_value   
> nulls_ratio avg_length  avg_frequency   hist_size   hist_type 
>   histogram
> +test t1  a   1   4   0.  4.  2.5000  0   NULL
> NULL
> +test t1  b   2   10  0.  4.  1.  0   NULL
> NULL
> +analyze
> +select * from t1 where a = 1 and b=3;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1SIMPLE  t1  ALL NULLNULLNULLNULL10  10.00   
> 2.7810.00   Using where
> +#
> +# with use_stat_tables= PREFERABLY_FOR_QUERIES
> +# analyze table t1 will  collect statistics if we use PERSISTENT
> +# for columns, indexes or everything
> +#
> +analyze table t

Re: [Maria-developers] Extending storage engine API for random-row extraction for histogram collection (and others)

[Sergey Petrunia]

On Tue, Dec 11, 2018 at 12:34:12AM +0200, Vicențiu Ciorbaru wrote:
> Hi!
> 
> Here is my proposal on extending the storage engine API to provide a
> functionality for retrieving random rows from tables (those that have
> indexes). The storage engines for which I plan to implement this are:
> MyISAM, Aria, Innodb. Possibly RocksDB, TokuDB.

Observations:

- as far as I understand, random skip scan is not possible with this API?
(which is probably fine as we expect that sampling will only retrieve a small
fraction of table rows, which means the difference between a forward-walking
skip scan and genuinely random probing is negligible).

- Can the scan return the same row twice?

- Do we want/need a concept of random "seed" which will cause the same rows to
be returned on the same table?

> 
> --- a/sql/handler.h
> +++ b/sql/handler.h
> @@ -2927,7 +2927,7 @@ class handler :public Sql_alloc
>/** Length of ref (1-8 or the clustered key length) */
>uint ref_length;
>FT_INFO *ft_handler;
> -  enum init_stat { NONE=0, INDEX, RND };
> +  enum init_stat { NONE=0, INDEX, RND, RANDOM };
>init_stat inited, pre_inited;
> 
> +  virtual int ha_random_sample_init() __attribute__((warn_unused_result))
> +  {
> +DBUG_ENTER("ha_random_sample_init");
> +inited= RANDOM;
> +DBUG_RETURN(random_sample_init());
> +  }
> +  virtual int ha_random_sample(uint inx,
> +   key_range *min_key,
> +   key_range *max_key)
> +__attribute__((warn_unused_result))
> +  {
> +DBUG_ENTER("ha_random_sample");
> +DBUG_ASSERT(inited == RANDOM);
> +DBUG_RETURN(random_sample(inx, min_key, max_key));
> +  }
> +  virtual int ha_random_sample_end() __attribute__((warn_unused_result))
> +  {
> +DBUG_ENTER("ha_random_sample_end");
> +inited= NONE;
> +DBUG_RETURN(random_sample_end());
> +  }
> +
> 
> This is the default implementation for a storage engine which does not
> support it:
> 
> +  virtual int random_sample_init() { return 0; } ;
> +  virtual int random_sample(uint idx, key_range *min_key, key_range
> *max_key)
> +  {
> +return HA_ERR_WRONG_COMMAND;
> +  }
> +  virtual int random_sample_end() { return 0; };
> 
> Alternative ideas: random_sample_init() takes the idx as a parameter and
> random_sample just fetches a row from the range using the index previously
> specified. The range can be left unspecified with nulls to provide a fetch
> from the full table range.
>  I don't know enough about storage engine internals to know if an index
> declaration within the init function instead of within the "sample"
> function is better. Maybe I am complicating it too much and a simple
> random_sample() function is sufficient, kind of how ha_records_in_range
> does it.
> 
> Thoughts?
> Vicențiu

-- 
BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] e1b0a8f9622: MDEV-6453: Assertion `inited==NONE || (inited==RND && scan)' failed in handler::ha_rnd_init(bool)

[Sergey Petrunia]

Hi Varun,

On Mon, Jan 29, 2018 at 12:27:35PM +0530, Varun wrote:
> revision-id: e1b0a8f9622ab8c2bab988cb71225f992fec320a 
> (mariadb-10.0.30-286-ge1b0a8f9622)
> parent(s): d01dbe66a8bf9cb6031f95159c49100f9299a768
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-01-29 12:23:31 +0530
> message:
> 
> MDEV-6453: Assertion `inited==NONE || (inited==RND && scan)' failed in 
> handler::ha_rnd_init(bool)
>with InnoDB, joins, AND/OR conditions
> 
> The inited parameter handler is not initialised when we do a quick_select 
> after a table scan.
> 

As far I understand, the problem only shows with "Range Checked for each 
record".
The failure scenario is as follows:
- range checked for each record plan in coonstructed
- the first check picks to a full table scan.
- the second check picks to do a QUICK_ROR_INTERSECT_SELECT scan
- QUICK_ROR_INTERSECT_SELECT starts to initialize the quick select.
- and it hits an assertion, because the handle object is already initialized
- index merge finds the handler not to be initialized correctly.


That is, the cause of handler object not being correctly initialized is "range
checked for each record" feature. 

Because of that, I think it should be fixed in that feature as well. A more
suitable location would be in range-checked-for-each-record's code, in
test_if_quick_select(): 

diff --git a/sql/sql_select.cc b/sql/sql_select.cc
index db3ed8a1aa9..6634554ee6a 100644
--- a/sql/sql_select.cc
+++ b/sql/sql_select.cc
@@ -19637,6 +19637,10 @@ test_if_quick_select(JOIN_TAB *tab)
 
   delete tab->select->quick;
   tab->select->quick=0;
+
+   if (tab->table->file->inited != handler::NONE)
+ tab->table->file->ha_index_or_rnd_end();
+
   int res= tab->select->test_quick_select(tab->join->thd, tab->keys,
   (table_map) 0, HA_POS_ERROR, 0,
   FALSE, /*remove where parts*/FALSE);

What do you think?  (If you agree, let's use the above variant)



> ---
>  mysql-test/r/range_innodb.result | 18 ++
>  mysql-test/t/range_innodb.test   | 17 +
>  sql/opt_range.cc |  2 ++
>  3 files changed, 37 insertions(+)
> 
> diff --git a/mysql-test/r/range_innodb.result 
> b/mysql-test/r/range_innodb.result
> index 794e6c7b3cc..8bb1c833a56 100644
> --- a/mysql-test/r/range_innodb.result
> +++ b/mysql-test/r/range_innodb.result
> @@ -37,3 +37,21 @@ id select_type table   typepossible_keys   key 
> key_len ref rowsExtra
>  1SIMPLE  t0  ALL NULLNULLNULLNULL10  
>  1SIMPLE  t2  range   a,b b   5   NULL201 Using 
> where; Using join buffer (flat, BNL join)
>  drop table t0,t1,t2;
> +CREATE TABLE t1 (
> +pk INT PRIMARY KEY, f1 INT, f2 CHAR(1), f3 CHAR(1),
> +KEY(f1), KEY(f2)
> +) ENGINE=InnoDB;
> +INSERT INTO t1 VALUES
> +(1,4,'v',NULL),(2,6,'v',NULL),(3,7,'c',NULL),(4,1,'e',NULL),(5,0,'x',NULL),
> +(6,7,'i',NULL),(7,7,'e',NULL),(8,1,'p',NULL),(9,7,'s',NULL),(10,1,'j',NULL),
> +(11,5,'z',NULL),(12,2,'c',NULL),(13,0,'a',NULL),(14,1,'q',NULL),(15,8,'y',NULL),
> +(16,1,'m',NULL),(17,1,'r',NULL),(18,9,'v',NULL),(19,1,'n',NULL);
> +CREATE TABLE t2 (f4 INT, f5 CHAR(1)) ENGINE=InnoDB;
> +INSERT INTO t2 VALUES (4,'q'),(NULL,'j');
> +SELECT * FROM t1 AS t1_1, t1 AS t1_2, t2
> +WHERE f5 = t1_2.f2 AND ( t1_1.f1 = 103 AND t1_1.f2 = 'o' OR t1_1.pk < f4 );
> +pk   f1  f2  f3  pk  f1  f2  f3  f4  f5
> +14   v   NULL14  1   q   NULL4   q
> +26   v   NULL14  1   q   NULL4   q
> +37   c   NULL14  1   q   NULL4   q
> +drop table t1,t2;
> diff --git a/mysql-test/t/range_innodb.test b/mysql-test/t/range_innodb.test
> index f76794814ef..605006587cc 100644
> --- a/mysql-test/t/range_innodb.test
> +++ b/mysql-test/t/range_innodb.test
> @@ -45,3 +45,20 @@ explain select * from t0 left join t2 on t2.a  t2.b between 50 and 250;
>  
>  drop table t0,t1,t2;
>  
> +CREATE TABLE t1 (
> +  pk INT PRIMARY KEY, f1 INT, f2 CHAR(1), f3 CHAR(1),
> +  KEY(f1), KEY(f2)
> +) ENGINE=InnoDB;
> +
> +INSERT INTO t1 VALUES
> +(1,4,'v',NULL),(2,6,'v',NULL),(3,7,'c',NULL),(4,1,'e',NULL),(5,0,'x',NULL),
> +(6,7,'i',NULL),(7,7,'e',NULL),(8,1,'p',NULL),(9,7,'s',NULL),(10,1,'j',NULL),
> +(11,5,'z',NULL),(12,2,'c',NULL),(13,0,'a',NULL),(14,1,'q',NULL),(15,8,'y',NULL),
> +(16,1,'m',NULL),(17,1,'r',NULL),(18,9,'v',NULL),(19,1,'n',NULL);
> +
> +CREATE TABLE t2 (f4 INT, f5 CHAR(1)) ENGINE=InnoDB;
> +INSERT INTO t2 VALUES (4,'q'),(NULL,'j');
> +
> +SELECT * FROM t1 AS t1_1, t1 AS t1_2, t2
> +WHERE f5 = t1_2.f2 AND ( t1_1.f1 = 103 AND t1_1.f2 = 'o' OR t1_1.pk < f4 );
> +drop table t1,t2;
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index f1d84e5c623..30b7f43ef28 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -3003,6 +3003,8 @@ int SQL_SELECT::test_quick_select(THD *thd, key_ma

Re: [Maria-developers] 1a4031d9c8f: MDEV-17297: stats.records=0 for a table of Archive engine when it has rows, when we run ANALYZE command

[Sergey Petrunia]

On Wed, Jan 23, 2019 at 04:11:58PM +0100, Sergei Golubchik wrote:
> On Nov 07, Sergei Petrunia wrote:
> > revision-id: 1a4031d9c8f314896daf8f7c19d6172b9bba313b 
> > (mariadb-10.3.10-49-g1a4031d9c8f)
> > parent(s): e058a251c10350f3727ca1df022dc5786933535b
> > author: Sergei Petrunia
> > committer: Sergei Petrunia
> > timestamp: 2018-11-07 11:58:30 +0300
> > message:
> > 
> > MDEV-17297: stats.records=0 for a table of Archive engine when it has rows, 
> > when we run ANALYZE command
> ... 
> > diff --git a/storage/archive/ha_archive.cc b/storage/archive/ha_archive.cc
> > index 487c0038239..9d7f7e66b28 100644
> > --- a/storage/archive/ha_archive.cc
> > +++ b/storage/archive/ha_archive.cc
> > @@ -1737,6 +1721,38 @@ int ha_archive::info(uint flag)
> >  }
> >  
> >  
> > +int ha_archive::external_lock(THD *thd, int lock_type)
> > +{
> > +  if (lock_type == F_RDLCK)
> > +  {
> > +// We are going to read from the table. Flush any pending writes that 
> > we
> > +// may have
> > +flush_and_clear_pending_writes();
> > +  }
> > +  return 0;
> > +}
> 
> This is somewhat unexpected. Normally buffers are flushed at the end of
> a statement, on F_UNLCK. For example, you'll have a better chance of
> surviving a crash (or kill -9).
> 
> Otherwise I don't see much of a difference, so do it as you like.

There is a difference. Consider a bulk load:
INSERT INTO archive_table ...;
INSERT INTO archive_table ...;
...

If we flush at statement end, the above fill do a flush after each statement.

If we flush in external_lock(F_RDLCK), the above will not flush, and
potentially execute faster.

> 
> Shouldn't it go into 5.5, as the earliest affected version?
> 

I assumed only security fixes should go into 5.5. If this fix is appropriate
for 5.5, I'm fine with pushing it there.

> > +void ha_archive::flush_and_clear_pending_writes()
> > +{
> > +  mysql_mutex_lock(&share->mutex);
> > +  if (share->dirty)
> > +  {
> > +DBUG_PRINT("ha_archive", ("archive flushing out rows for scan"));
> > +DBUG_ASSERT(share->archive_write_open);
> > +azflush(&(share->archive_write), Z_SYNC_FLUSH);
> > +share->dirty= FALSE;
> > +  }
> > +
> > +  /* 
> > +This should be an accurate number now, though bulk and delayed inserts 
> > can
> > +cause the number to be inaccurate.
> > +  */
> > +  stats.records= share->rows_recorded;
> > +  mysql_mutex_unlock(&share->mutex);
> > +}

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Some more input on optimizer_trace

[Sergey Petrunia]

Hi Varun,

I've did some adjustments MDEV-18489 and pushed the patch into
10.4-optimizer-trace, please check it out.

Also I have filed MDEV-18527 and MDEV-18528.

Some input on the code:

> --- 10.4-optimizer-trace-orig/sql/sql_select.cc
> +++ 10.4-optimizer-trace-cl/sql/sql_select.cc

> @@ -15983,12 +16250,26 @@ optimize_cond(JOIN *join, COND *conds,
>that occurs in a function set a pointer to the multiple equality
>predicate. Substitute a constant instead of this field if the
>multiple equality contains a constant.
> -*/ 
> -DBUG_EXECUTE("where", print_where(conds, "original", QT_ORDINARY););
> +*/
> +
> +Opt_trace_context *const trace = &thd->opt_trace;
> +Json_writer *writer= trace->get_current_json();
> +Json_writer_object trace_wrapper(writer);
> +Json_writer_object trace_cond(writer, "condition_processing");
> +trace_cond.add("condition", join->conds == conds ? "WHERE" : "HAVING")
> +  .add("original_condition", conds);
> +
> +Json_writer_array trace_steps(writer, "steps");
> +  DBUG_EXECUTE("where", print_where(conds, "original", QT_ORDINARY););

Small question: Why was DBUG_EXECUTE shifted right? 

A bigger question: the code seems unnecesarily verbose:

1. > +Opt_trace_context *const trace = &thd->opt_trace;
2. > +Json_writer *writer= trace->get_current_json();
3. > +Json_writer_object trace_wrapper(writer);
4. > +Json_writer_object trace_cond(writer, "condition_processing");

Can we save the space by just calling the constructors:

  Json_writer_object trace_wrapper(thd);
  Json_writer_object trace_cond(thd, "condition_processing");

?
This applies here and in many other places.

Alternative, we could use:

  Json_writer_object trace_wrapper(thd);
  Json_writer_object trace_cond(trace_wrapper, "condition_processing");

.. which makes the nesting clearer (and we could also add debug safety check: it
is invalid to operate on trace_wrapper until trace_cond hasn't been end()'ed)


> --- 10.4-optimizer-trace-orig/sql/opt_range.cc
> +++ 10.4-optimizer-trace-cl/sql/opt_range.cc  
> 
> +void TRP_ROR_UNION::trace_basic_info(const PARAM *param,
> + Json_writer_object *trace_object) const
> +{
> +  Opt_trace_context *const trace = ¶m->thd->opt_trace;
> +  Json_writer* writer= trace->get_current_json();
> +  trace_object->add("type", "index_roworder_union");
> +  Json_writer_array ota(writer, "union_of");

The name 'ota' makes sense in MySQL codebase (where it is a contraction of
Optimizer_trace_array), but is really confusing in MariaDB codebase. Please
change everywhere to "smth_trace" or something like that (jwa in the worst
case).

> @@ -2654,12 +2833,18 @@ int SQL_SELECT::test_quick_select(THD *t
> 
>  if (cond)
>  {
> -  if ((tree= cond->get_mm_tree(¶m, &cond)))
> +  {
> +Json_writer_array trace_range_summary(writer,
> +   "setup_range_conditions");
> +tree= cond->get_mm_tree(¶m, &cond);
> +  }

Does this ever produce anything meaningful, other than empty:

"setup_range_conditions": [],

In MySQL, the possible contents of this array are:

  "impossible_condition": {
"cause": "comparison_with_null_always_false"
  } /* impossible_condition */

   "impossible_condition": {
 "cause": "null_field_in_non_null_column"
   } /* impossible_condition */

But in MariaDB we don't have this (should we?)

Also,  why_use_underscores_in_value_of_cause?  It is a quoted string where
spaces are allowed. MySQL seems to have figured this out at some point and have
a few cause strings using spaces.

> --- 10.4-optimizer-trace-orig/sql/opt_table_elimination.cc
> +++ 10.4-optimizer-trace-cl/sql/opt_table_elimination.cc  
> @@ -522,7 +524,8 @@ eliminate_tables_for_list(JOIN *join,
>List *join_list,
>table_map tables_in_list,
>Item *on_expr,
> -  table_map tables_used_elsewhere);
> +  table_map tables_used_elsewhere,
> +  Json_writer_array* eliminate_tables);

Please change the name to something indicating it's just a trace.


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] Some more input on optimizer_trace

[Sergey Petrunia]

On Sun, Feb 10, 2019 at 02:14:51PM +0200, Sergey Petrunia wrote:
> Hi Varun,
> 
> I've did some adjustments MDEV-18489 and pushed the patch into
> 10.4-optimizer-trace, please check it out.
> 
> Also I have filed MDEV-18527 and MDEV-18528.
> 
> Some input on the code:
> 
> > --- 10.4-optimizer-trace-orig/sql/sql_select.cc
> > +++ 10.4-optimizer-trace-cl/sql/sql_select.cc
> 
One more thing - Json_value_context is not really a context anymore.

(the original intent was that "one gets a value context object if the
Json_writer's current state is such that it currently expects a value (and not
a name)"  but apparently it is not used this way. Which is fine, but then I
guess the object should be renamed. (to _helper? or something like that?)

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

[Maria-developers] Input on Optimizer trace code, part #2

[Sergey Petrunia]

Hi Varun,

Please find more input below.

> --- 10.4-optimizer-trace-orig/sql/sql_derived.cc  2019-02-10 
> 14:25:59.217015365 +0300
> +++ 10.4-optimizer-trace-cl/sql/sql_derived.cc2019-02-12 
> 14:04:05.446571305 +0300
> @@ -486,6 +486,24 @@ exit_merge:
>DBUG_RETURN(res);
>  
>  unconditional_materialization:
> +
> +  if (unlikely(thd->trace_started()))
> +  {
> +/*
> + Add to the optimizer trace the change in choice for merged
> + derived tables/views to materialised ones.
> +*/
> +Json_writer_object trace_wrapper(thd);
> +Json_writer_object trace_derived(thd, derived->is_derived() ?
> +   "derived" : "view");
> +trace_derived.add("table", derived->alias.str ? derived->alias.str : 
> "")
> + .add_select_number(derived->get_unit()->
> +first_select()->select_number)
> + .add("initial_choice", "merged")
> + .add("final_choice", "materialized")
> + .add("cause", cause);

Please add test coverage for this (I cannot find it in the testsuite).


> +select * from v1 {
> +  "steps": [
> +{
> +  "join_preparation": {
> +"select_id": 1,
> +"steps": [
> +  {
> +"view": {
> +  "table": "v1",
> +  "select_id": 2,
> +  "merged": true

The above looks a bit odd, "view", inside it "table", and then either
"materilized" or "merged". I would prefer 
  "algorithm": "materialize"
or
  "algorithm": "merge"

> diff -urpN '--exclude=.*' 10.4-optimizer-trace-orig/sql/sql_select.cc 
> 10.4-optimizer-trace-cl/sql/sql_select.cc
> --- 10.4-optimizer-trace-orig/sql/sql_select.cc   2019-02-10 
> 14:25:59.225015425 +0300
> +++ 10.4-optimizer-trace-cl/sql/sql_select.cc 2019-02-12 14:04:05.450571322 
> +0300
> @@ -345,6 +349,40 @@ bool dbug_user_var_equals_int(THD *thd,
>  }
>  #endif
>  
> +static void trace_table_dependencies(THD *thd,
> + JOIN_TAB *join_tabs, uint table_count)
> +{
> +  Json_writer_object trace_wrapper(thd);
> +  Json_writer_array trace_dep(thd, "table_dependencies");
> +  for (uint i = 0; i < table_count; i++)
> +  {
> +TABLE_LIST *table_ref = join_tabs[i].tab_list;
> +Json_writer_object trace_one_table(thd);
> +trace_one_table.add_table_name(&join_tabs[i]);
> +trace_one_table.add("row_may_be_null",
> +   (bool)table_ref->table->maybe_null);
> +const table_map map = table_ref->get_map();
> +DBUG_ASSERT(map < (1ULL << table_count));
> +for (uint j = 0; j < table_count; j++)
> +{
> +  if (map & (1ULL << j))
> +  {
> +trace_one_table.add("map_bit", static_cast(j));
> +break;
> +  }
> +}
> +Json_writer_array depends_on(thd, "depends_on_map_bits");
> +static_assert(sizeof(table_ref->get_map()) <= 64,
> +  "RAND_TABLE_BIT may be in join_tabs[i].dependent, so we 
> test "
> +  "all 64 bits.");
sizeof() is in bytes, not bits, so the above assert actually checks for
whether table_map is 64 bytes large (which is incorrect).

I would suggest to switch to using Table_map_iterator and forget about the
64-byte limitation (and the value of RAND_TABLE_BIT, too)

> +for (uint j = 0; j < 64; j++)
> +{
> +  if (join_tabs[i].dependent & (1ULL << j))
> +depends_on.add(static_cast(j));
> +}

> --- 10.4-optimizer-trace-orig/sql/opt_table_elimination.cc2019-02-10 
> 14:25:59.209015304 +0300
> +++ 10.4-optimizer-trace-cl/sql/opt_table_elimination.cc  2019-02-12 
> 14:04:05.442571289 +0300
> @@ -522,7 +524,8 @@ eliminate_tables_for_list(JOIN *join,
>List *join_list,
>table_map tables_in_list,
>Item *on_expr,
> -  table_map tables_used_elsewhere);
> +  table_map tables_used_elsewhere,
> +  Json_writer_array* eliminate_tables);

I think I wrote this already - I think 'eliminate_tables' is a very poor name
for the argument-  it doesn't show that this is just a trace output object. If
something is named 'eliminate_tables' I would expect it to play some active
role (e.g. being a list of tables we are trying to eliminate or something like
that).
Please change to something like trace, trace_eliminated, trace_for_eliminated,
...

> @@ -2340,8 +2377,13 @@ int pull_out_semijoin_tables(JOIN *join)
>  bool optimize_semijoin_nests(JOIN *join, table_map all_table_map)
>  {
>DBUG_ENTER("optimize_semijoin_nests");
> +  THD *thd= join->thd;
>List_iterator sj_list_it(join->select_lex->sj_nests);
>TABLE_LIST *sj_nest;
> +  Json_writer_object wrapper(thd);
> +  Json_writer_object trace_semijoin_nest(thd,
> +  
> "execution_plan_for_potential_materialization");
> +  Json_writer_array trace_steps_array(thd, "steps");
> 

Re: [Maria-developers] [Commits] 500d909: MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly

[Sergey Petrunia]

> revision-id: 500d909a795022255fb6f6c44cd310636afc0b5a 
> (mariadb-10.3.6-226-g500d909)
> parent(s): 88d89ee0bae24b71416c2af4f4c2f2be7b6a033a
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2019-03-11 20:20:35 +0530
> message:
> 
> MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
> 
> Keys with multi-key parts were not being printed correctly, only the first 
> key part
> was getting printed.
> Fixed it by making sure append_range_all_keyparts function is called for the 
> remaining keyparts.
>
...

> diff --git a/mysql-test/main/opt_trace.result 
> b/mysql-test/main/opt_trace.result
> index 4c3e2b3..a4c2b86 100644
> --- a/mysql-test/main/opt_trace.result
> +++ b/mysql-test/main/opt_trace.result
> @@ -6022,3 +6022,428 @@ t_outer_2.a in (select t_inner_3.a from t2 t_inner_3, 
> t1 t_inner_4)   {
>  set @@optimizer_switch= @save_optimizer_switch;
>  drop table t1,t2;
>  set optimizer_trace='enabled=off';
> +#
> +# MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
> +#
> +create table t0(a int);
> +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
> +create table one_k (a int);
> +insert into one_k select A.a + B.a*10 + C.a*100 from t0 A, t0 B, t0 C;
> +create table t1 ( a int, b int, c int, d int, key (a,b), key(c,d));
> +insert into t1 select a,a,a,a from one_k;
> +set optimizer_trace=1;
> +explain format=json select * from t1 where a > 10 and b < 10 and c=0 and d=1;
...
> +select * from INFORMATION_SCHEMA.OPTIMIZER_TRACe;

This produces a lot of irrelevant output. Can you use something like

  select JSON_DETAILED(JSON_EXTRACT(a, '$**.range_analysis')) from 
optimizer_trace

to select on the part of the trace that we need?

> +  "analyzing_range_alternatives": {
> +"range_scan_alternatives": [
> +  {
> +"index": "a",
> +"ranges": ["10 < a AND NULL < b < 10"],

This is not a valid range. If one forces this quick select to be picked for the
query plan, they can look into .trace and see the ranges:

explain format=json select * from t1 force index (a) where a > 10 and b < 10;

T@19   : | | | | | | | | | | >print_quick
quick range select, key a, length: 5
  10 < X
other_keys: 0x0:
T@19   : | | | | | | | | | | =10 to see a different kind of range being produced).

(One could argue that what is printed is a representation of the SEL_ARG graph,
but in that case it is extremely confusing. I think, we should start with
showing ranges, not SEL_ARG graphs).

I think, we should not try to intrepret the SEL_ARG graph ourselves but rather 
use sel_arg_range_seq_init / sel_arg_range_seq_next to produce ranges that can
be printed.  Any objections to this?

> +"rowid_ordered": false,
> +"using_mrr": false,
> +"index_only": false,
> +"rows": 989,
> +"cost": 1272.8,
> +"chosen": false,
> +"cause": "cost"
> +  },
> +  {
> +"index": "c",
> +"ranges": ["0 <= c <= 0 AND 1 <= d <= 1"],
> +"rowid_ordered": true,
> +"using_mrr": false,
> +"index_only": false,
> +"rows": 1,
> +"cost": 2.3783,
> +"chosen": true
> +  }
> +],


BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] e0fa1993692: MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value

[Sergey Petrunia]

Hi Varun,

First, Some input on the test:

On Wed, Mar 27, 2019 at 01:55:16PM +0530, Varun wrote:
> revision-id: e0fa1993692a3552feddae56cbee3078d4da2121 
> (mariadb-10.2.22-91-ge0fa1993692)
> parent(s): 50a8fc52988d13a5164a1a542b9d7a85e3ecc1c1
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2019-03-27 13:48:44 +0530
> message:
> 
> MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value
> 
> To fix the crash there we need to make sure that the
> server while storing the statistical values in statistical tables should do it
> in a multi-byte safe way.
> Also there is no need to throw warnings if there is truncation while storing
> values from statistical fields.
> 
...
> --- a/mysql-test/r/stat_tables_innodb.result
> +++ b/mysql-test/r/stat_tables_innodb.result
> @@ -651,6 +651,63 @@ SELECT MAX(pk) FROM t1;
>  MAX(pk)
>  NULL
>  DROP TABLE t1;
> +#
> +# MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value
> +#
> +set names utf8;
> +set 
> @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity;
> +set optimizer_use_condition_selectivity=4;
> +set use_stat_tables=preferably;
> +set @save_histogram_size= @@histogram_size;
> +set histogram_size=255;
> +create table t1 ( a varchar(255) character set utf8);
> +insert into t1 values (REPEAT('ӥ',255)), (REPEAT('ç',255));
> +analyze table t1;
> +TableOp  Msg_typeMsg_text
> +test.t1  analyze status  Engine-independent statistics collected
> +test.t1  analyze status  OK
> +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats;
> +HEX(RIGHT(min_value, 1)) length(min_value)
> +A7   254
> +select HEX(RIGHT(max_value, 1)), length(max_value) from mysql.column_stats;
> +HEX(RIGHT(max_value, 1)) length(max_value)
> +A5   254
> +analyze select * from t1 where a  >= 'ӥ';
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1SIMPLE  t1  ALL NULLNULLNULLNULL2   2.00
> 50.00   50.00   Using where
> +set @save_sql_mode= @@sql_mode;
> +set 
> sql_mode='ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
> +update mysql.column_stats set min_value= REPEAT('ӥ',255);

This looks scary, what if there were some other data in these tables? 
Can you change add a WHERE db_name=database() and table_name='t1' ? 

> +Warnings:
> +Warning  1265Data truncated for column 'min_value' at row 1
> +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats;
> +HEX(RIGHT(min_value, 1)) length(min_value)
> +D3   255
> +analyze select * from t1 where a  >= 'ӥ';
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1SIMPLE  t1  ALL NULLNULLNULLNULL2   2.00
> 50.00   50.00   Using where
> +set names latin1;
> +drop table t1;
> +CREATE TABLE t1 (col1 date);
> +INSERT INTO t1 VALUES('2004-01-01'),('2004-02-29');
> +INSERT INTO t1 VALUES('-10-31');
> +analyze table t1;
> +TableOp  Msg_typeMsg_text
> +test.t1  analyze status  Engine-independent statistics collected
> +test.t1  analyze status  OK
> +update mysql.column_stats set min_value='2004-0-31123';

The same as above.

> +select min_value from mysql.column_stats;
> +min_value
> +2004-0-31123

This case also makes one wonder whether not producing a warning was a good idea
after all.

A varchar string that is 255-byte long, and after 255 bytes it gets truncated
in the middle of a character is a rare case.

An incorrect datetime value... not as rare?

> +select * from t1;

Is the above supposed to read the EITS data? It doesn't as the query doesn't
use the WHERE clause.
> +col1
> +2004-01-01
> +2004-02-29
> +-10-31
> +drop table t1;
> +set @@sql_mode= @save_sql_mode;
>  set use_stat_tables=@save_use_stat_tables;
> +set @@histogram_size= @save_histogram_size;
> +set 
> @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity;
>  set optimizer_switch=@save_optimizer_switch_for_stat_tables_test;
>  SET SESSION STORAGE_ENGINE=DEFAULT;
> diff --git a/mysql-test/t/stat_tables.test b/mysql-test/t/stat_tables.test
> index b89ab2bbd2d..114c5c97a7e 100644
> --- a/mysql-test/t/stat_tables.test
> +++ b/mysql-test/t/stat_tables.test
> @@ -401,4 +401,44 @@ SELECT MAX(pk) FROM t1;
>  
>  DROP TABLE t1;
>  
> +--echo #
> +--echo # MDEV-18899: Server crashes in 
> Field::set_warning_truncated_wrong_value
> +--echo #
> +
> +set names utf8;
> +set 
> @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity;
> +set optimizer_use_condition_selectivity=4;
> +set use_stat_tables=preferably;
> +set @save_histogram_size= @@histogram_size;
> +set histogram_size=255;
> +
> +create table t1 ( a varchar(255) character set utf8);
> +insert into t1 values (REPEAT('ӥ',255)), (REPEAT('ç',255));
>

[Maria-developers] EITS member variables in both TABLE and TABLE_SHARE?

[Sergey Petrunia]

Hello Igor,

TABLE_SHARE has TABLE_STATISTICS_CB, which has these members:
struct TABLE_STATISTICS_CB
{
  ...
  bool stats_is_read;/* Statistical data for table has been read
from statistical tables */
  bool histograms_are_read;
}

At the same time, TABLE has:

struct TABLE
{
  ...
  bool stats_is_read; /* Persistent statistics is read for the table */
  bool histograms_are_read;

I would like to understand why both structures need these variables? Is there
any difference in meaning between them?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] a1929d001c7: MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly

[Sergey Petrunia]

Hi Varun,

In general, the patch moves in the right direction, but it is not yet there. 

Consider this example:

create table ten(a int);
insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
create table t1 (
  a int not null,
  b int not null,
  c int not null,
  d int not null,
  key(a,b,c)
);

insert into t1 select a,a, a,a from ten;

set optimizer_trace=1;
explain select * from t1 where a between 1 and 4 and b < 50;

mysql> select
-> JSON_DETAILED(JSON_EXTRACT(trace, '$**.analyzing_range_alternatives'))
-> from INFORMATION_SCHEMA.OPTIMIZER_TRACE;

{
"range_scan_alternatives": 
[

{
"index": "a",
"ranges": 
[
"(a) < (4,50)"
],
"rowid_ordered": false,
"using_mrr": false,
"index_only": false,
"rows": 1,
"cost": 2.5002,
"chosen": true
}
],
"analyzing_roworder_intersect": 
{
"cause": "too few roworder scans"
},
"analyzing_index_merge_union": 
[
]
}
] |

while print_quick produces:

T@18   : | | | | | | | | | | >print_quick
quick range select, key a, length: 8
  1 <= X < 4/50
other_keys: 0x0:
T@18   : | | | | | | | | | |  revision-id: a1929d001c7b25e26d178834a196020bade819b8 
> (mariadb-10.3.6-318-ga1929d001c7)

Any idea why does the commit trigger show "mariadb-10.3.." ? The patch is
against 10.4, isnt it?

> parent(s): 0dc442ac61ac7ff1a1d6bd7d6090c0f2251fb558
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2019-04-03 13:10:07 +0530
> message:
> 
> MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
> 
> Changed the function append_range_all_keyparts to use sel_arg_range_seq_init 
> / sel_arg_range_seq_next to produce ranges.
> Also adjusted to print format for the ranges, now the ranges are printed as:
> (keypart1_min, keypart2_min,..)  OP (keypart1_name,keypart2_name, ..) OP 
> (keypart1_max,keypart2_max, ..)
> 
> Also added more tests for range and index merge access for optimizer trace

...
> diff --git a/sql/opt_range.cc b/sql/opt_range.cc
> index 5ab3d70214d..f53efc55158 100644
> --- a/sql/opt_range.cc
> +++ b/sql/opt_range.cc
> @@ -431,16 +431,18 @@ static int and_range_trees(RANGE_OPT_PARAM *param,
>  static bool remove_nonrange_trees(RANGE_OPT_PARAM *param, SEL_TREE *tree);
>  
>  static void print_key_value(String *out, const KEY_PART_INFO *key_part,
> -const uchar *key);
> +const uchar* key, uint length);
> +static void print_keyparts_name(String *out, const KEY_PART_INFO *key_part,
> + uint n_keypart, key_part_map keypart_map);
Please fix indentation ^

>  
>  static void append_range_all_keyparts(Json_writer_array *range_trace,
> -  String *range_string,
> -  String *range_so_far, const SEL_ARG 
> *keypart,
> +  PARAM param, uint idx,
> +  SEL_ARG *keypart,
>const KEY_PART_INFO *key_parts);
>  
>  static
> -void append_range(String *out, const KEY_PART_INFO *key_parts,
> -  const uchar *min_key, const uchar *max_key, const uint 
> flag);
> +void append_range(String *out, const KEY_PART_INFO *key_part,
> +  KEY_MULTI_RANGE *range, uint n_key_parts);
Please fix indentation ^
>  
>  
>  /*
> @@ -2273,10 +2275,7 @@ void TRP_RANGE::trace_basic_info(const PARAM *param,
>// TRP_RANGE should not be created if there are no range intervals
>DBUG_ASSERT(key);
>  
> -  String range_info;
> -  range_info.length(0);
> -  range_info.set_charset(system_charset_info);
> -  append_range_all_keyparts(&trace_range, NULL, &range_info, key, key_part);
> +  append_range_all_keyparts(&trace_range, *param, key_idx, key, key_part);
>  }
>  
>  
> @@ -2489,10 +2488,8 @@ void TRP_GROUP_MIN_MAX::trace_basic_info(const PARAM 
> *param,
>// can have group quick without ranges
>if (index_tree)
>{
> -String range_info;
> -range_info.set_charset(system_charset_info);
> -append_range_all_keyparts(&trace_range, NULL, &range_info, index_tree,
> -  key_part);
> +append_range_all_keyparts(&trace_range, *param, param_idx,
> +  index_tree, key_part);
>}
>  }
>  
> @@ -6398,20 +6395,9 @@ void TRP_ROR_INTERSECT::trace_basic_info(const PARAM 
> *param,
>  trace_isect_idx.add("rows", (*cur_scan)->records);
>  
>  Json_writer_array trace_range(thd, "ranges");
> -for (const SEL_ARG *current= (*cur_scan)->sel_arg->first(); current;
> - current= current->next)
> -{
> -  String range_info;
> -  range_info.set_cha

[Maria-developers] MDEV-18822: possible slowdown with optimizer trace

[Sergey Petrunia]

Hi Varun,

So I was thinking of the impact of the optimizer trace, and decided to look at
how the most common check "if optimizer trace is enabled" is done.

The code at SQL layer does this kind of check 

  if (thd->trace_started())

(it is wrapped somewhere with "unlikely" but this is not consistent)


THD::trace_started is defined as:


bool THD::trace_started()
{
  return opt_trace.is_started();
}

which opt_trace here is Opt_trace_context. looking there:

class Opt_trace_context 
{
  ...
  bool is_started()
  {
return current_trace && is_enabled();
  }

following:

bool Opt_trace_context::is_enabled()
{
  if (current_trace)
return current_trace->is_enabled();
  return false;
}

current_trace is Opt_trace_stmt, looking there:

bool Opt_trace_stmt::is_enabled()
{
  return I_S_disabled == 0;
}

This is quite convoluted, and could probably made shorter.

Disassembling sql_select.cc, I can see that the calls to thd->trace_started()
are not inlined:

   if (unlikely(thd->trace_started()))
   10ac9:   4c 89 ffmovrdi,r15
   10acc:   e8 00 00 00 00  call   10ad1 
<_ZL32best_extension_by_limited_searchP4JOINyjddjjj+0x1a1>
   10ad1:   84 c0   test   al,al
   10ad3:   0f 85 fc 02 00 00   jne10dd5 
<_ZL32best_extension_by_limited_searchP4JOINyjddjjj+0x4a5>


if (unlikely(thd->trace_started()))
   1197b:   4c 89 f7movrdi,r14
   1197e:   e8 00 00 00 00  call   11983 
<_Z11choose_planP4JOINy+0x8a3>
   11983:   84 c0   test   al,al
   11985:   0f 85 2f 01 00 00   jne11aba 
<_Z11choose_planP4JOINy+0x9da>

  if (thd->trace_started())
   2d6f7:   48 8b bd b0 fc ff ffmovrdi,QWORD PTR [rbp-0x350]
   2d6fe:   e8 00 00 00 00  call   2d703 
<_ZN4JOIN14optimize_innerEv+0x1963>
   2d703:   84 c0   test   al,al
   2d705:   0f 85 6c 09 00 00   jne2e077 
<_ZN4JOIN14optimize_innerEv+0x22d7>

if (thd->trace_started())
   2d7da:   48 8b bd b0 fc ff ffmovrdi,QWORD PTR [rbp-0x350]
   2d7e1:   e8 00 00 00 00  call   2d7e6 
<_ZN4JOIN14optimize_innerEv+0x1a46>
   2d7e6:   84 c0   test   al,al
   2d7e8:   0f 85 ff 1a 00 00   jne2f2ed 
<_ZN4JOIN14optimize_innerEv+0x354d>

But does it all matter? 

I think the action plan should be:

1. Reproduce the slowdown as it was reported in MDEV-18822. 
2. If #1 succeeds in several re-runs, declare THD::trace_started as:

class THD {
public:
  bool trace_started; // initialize this to false in constructor and never 
touch it 
  inline bool trace_started() const { return trace_started; }
};
 
and then use search-replace to change "thd->trace_started()"  to 
"unlikely(thd->trace_started)" 

and then re-run the benchmark.

Any thoughts?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 167a96b5157: MDEV-9959: A serious MariaDB server performance bug

[Sergey Petrunia]

On Tue, Mar 27, 2018 at 04:48:10PM +0530, Varun wrote:
> revision-id: 167a96b5157049408a6ad4bca7abcd376af93fb5 
> (mariadb-10.3.0-644-g167a96b5157)
> parent(s): 4359c6b4806605c78987e50cab3a6b42016b7603
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-03-27 16:45:46 +0530
> message:
> 
> MDEV-9959: A serious MariaDB server performance bug
> 
> step#1: if a derived table has SELECT DISTINCT, provide index statistics for 
> it so that the join optimizer in the
> upper select knows that ref access to the table will produce one row.
> 
> Added handling for multiple selects in the derived table

...
> diff --git a/mysql-test/r/mdev9959.result b/mysql-test/r/mdev9959.result
> new file mode 100644
> index 000..049e0350cca
> --- /dev/null
> +++ b/mysql-test/r/mdev9959.result
> @@ -0,0 +1,46 @@
> +create table t1(a int);
> +insert into t1 values (1),(2),(3),(4),(5),(6);
> +create table t2(a int, b int,c int);
> +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5);
> +create table t3(a int, b int);
> +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2);
> +table "" should have type=ref and rows=1
> +one select in derived table
> +with distinct
> +analyze select * from t1 , ((select distinct t2.a from t2 order by c))q  
> where t1.a=q.a;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1PRIMARY t1  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using where
> +1PRIMARY   ref key0key05   test.t1.a   
> 1   1.00100.00  100.00  
> +2DERIVED t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using temporary; Using filesort
> +# multiple selects in derived table
> +# NO UNION ALL
> +analyze select * from t1 , ( (select t2.a,t2.b from t2 order by c) union  
> (select t2.a,t2.b from t2 order by c))q  where t1.a=q.a;
> +id   select_type table   typepossible_keys   key key_len ref 
> rowsr_rows  filteredr_filtered  Extra
> +1PRIMARY t1  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  Using where
> +1PRIMARY   ref key0key05   test.t1.a   
> 1   1.00100.00  100.00  
> +2DERIVED t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  
> +3UNION   t2  ALL NULLNULLNULLNULL6   6.00
> 100.00  100.00  
> +NULL UNION RESULT  ALL NULLNULLNULLNULL
> NULL6.00NULLNULL

How would you explain the above? The derived table is a union of two selects,
each of which produces 6 rows. Neither of the selects have DISTINCT attribute.

Why does the optimization fire and set ref=1 in this case?

(if both parts of UNION had distinct, this would make sense. But if neither
does, I don't see any logic)

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Maria-developers] [Commits] 167a96b5157: MDEV-9959: A serious MariaDB server performance bug

[Sergey Petrunia]

On Tue, Mar 27, 2018 at 04:48:10PM +0530, Varun wrote:
> revision-id: 167a96b5157049408a6ad4bca7abcd376af93fb5 
> (mariadb-10.3.0-644-g167a96b5157)
> parent(s): 4359c6b4806605c78987e50cab3a6b42016b7603
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2018-03-27 16:45:46 +0530
> message:
> 
> MDEV-9959: A serious MariaDB server performance bug
> 
> step#1: if a derived table has SELECT DISTINCT, provide index statistics for 
> it so that the join optimizer in the
> upper select knows that ref access to the table will produce one row.
> 
> Added handling for multiple selects in the derived table
> 
> ---
>  mysql-test/r/cte_nonrecursive.result|  8 +--
>  mysql-test/r/cte_recursive.result   |  6 +-
>  mysql-test/r/derived_cond_pushdown.result   | 82 
> -
>  mysql-test/r/derived_view.result|  2 +-
>  mysql-test/r/join_cache.result  |  6 +-
>  mysql-test/r/mdev9959.result| 46 ++
>  mysql-test/r/subselect_extra.result |  2 +-
>  mysql-test/r/subselect_extra_no_semijoin.result |  2 +-
>  mysql-test/t/mdev9959.test  | 25 
>  sql/sql_lex.h   |  2 +
>  sql/sql_union.cc| 60 ++
>  sql/sql_yacc.yy |  3 +
>  sql/table.cc| 19 ++
>  13 files changed, 209 insertions(+), 54 deletions(-)
> 
> diff --git a/mysql-test/r/derived_cond_pushdown.result 
> b/mysql-test/r/derived_cond_pushdown.result
> index 32d3c88cc8d..3723e25a494 100644
> --- a/mysql-test/r/derived_cond_pushdown.result
> +++ b/mysql-test/r/derived_cond_pushdown.result
> @@ -5187,7 +5187,7 @@ explain select * from v2_union as v,t2 where
>  ((v.a=6) or (v.a=8)) and (v.c>200) and (v.a=t2.a);
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY t2  ALL NULLNULLNULLNULL9   Using 
> where
> -1PRIMARY   ref key0key05   test.t2.a   
> 6   Using where
> +1PRIMARY   ref key0key05   test.t2.a   
> 1   Using where

As agreed on the call: need to check what is the cause of this. Here, the temp
table has distinct rows, that is, {a,b,c} are distinct. But ref access only
uses the first component, where does rows=1 come from? (if this is how
best_access_path computes an estimate for prefix when it only has the estimate
for the full key ... fine)


>  2DERIVED t1  ALL NULLNULLNULLNULL20  Using 
> where; Using temporary; Using filesort
>  3UNION   t1  ALL NULLNULLNULLNULL20  Using 
> where; Using temporary; Using filesort
>  4UNION   t1  ALL NULLNULLNULLNULL20  Using 
> where; Using temporary; Using filesort
> @@ -5213,7 +5213,7 @@ EXPLAIN
>"key_length": "5",
>"used_key_parts": ["a"],
>"ref": ["test.t2.a"],
> -  "rows": 6,
> +  "rows": 1,
>"filtered": 100,
>"attached_condition": "v.c > 200",
>"materialized": {
> @@ -5358,7 +5358,7 @@ a   b   c   a   b   c   d
>  explain select * from v3_union as v,t2 where (v.a=t2.a) and (v.c>6);
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY t2  ALL NULLNULLNULLNULL9   Using 
> where
> -1PRIMARY   ref key0key05   test.t2.a   
> 4   Using where
> +1PRIMARY   ref key0key05   test.t2.a   
> 1   Using where
>  2DERIVED t1  ALL NULLNULLNULLNULL20  Using 
> where
>  3UNION   t1  ALL NULLNULLNULLNULL20  Using 
> where
>  NULL UNION RESULT  ALL NULLNULLNULLNULL
> NULL
> @@ -5382,7 +5382,7 @@ EXPLAIN
>"key_length": "5",
>"used_key_parts": ["a"],
>"ref": ["test.t2.a"],
> -  "rows": 4,
> +  "rows": 1,
>"filtered": 100,
>"attached_condition": "v.c > 6",
>"materialized": {
> @@ -5476,7 +5476,7 @@ a   b   c   a   b   c   d
>  explain select * from v3_union as v,t2 where (v.a=t2.a) and ((t2.a>1) or 
> (v.b<20));
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY t2  ALL NULLNULLNULLNULL9   Using 
> where
> -1PRIMARY   ref key0key05   test.t2.a   
> 4   Using where
> +1PRIMARY   ref key0key05   test.t2.a   
> 1   Using where
>  2DERIVED t1  ALL NULLNULLNULLNULL20  Using 
> where
>  3UNION   t1  ALL NULLNULLNULLNULL20  Using 
> where
>  NULL UNION RESULT  ALL NULLNULLN

Re: [Maria-developers] [Commits] 3032cd8e91f: step #1: if a derived table has SELECT DISTINCT, provide index statistics for it so that the join optimizer in the

[Sergey Petrunia]

Hi Varun,

Please mention the MDEV# in the commit comment.

Otherwise, the patch is ok to push.

On Mon, Apr 22, 2019 at 07:09:45PM +0530, Varun wrote:
> revision-id: 3032cd8e91f1e1ead8b6f941e75cd29e473e7eaa 
> (mariadb-10.3.10-283-g3032cd8e91f)
> parent(s): f4019f5b3544a18f3ddf32df2c5214c3f8dabdce
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2019-04-22 18:19:25 +0530
> message:
> 
> step #1: if a derived table has SELECT DISTINCT, provide index statistics for 
> it so that the join optimizer in the
> upper select knows that ref access to the table will produce one row.
> 
> ---
>  mysql-test/main/cte_nonrecursive.result|  8 ++--
>  mysql-test/main/derived.result | 54 
> ++
>  mysql-test/main/derived.test   | 30 
>  mysql-test/main/derived_view.result|  2 +-
>  mysql-test/main/subselect_extra.result |  2 +-
>  mysql-test/main/subselect_extra_no_semijoin.result |  2 +-
>  sql/sql_lex.h  |  1 +
>  sql/sql_union.cc   | 40 
>  sql/table.cc   | 20 
>  9 files changed, 152 insertions(+), 7 deletions(-)
> 
> diff --git a/mysql-test/main/cte_nonrecursive.result 
> b/mysql-test/main/cte_nonrecursive.result
> index b846ec2d8ac..d80d34ecc7f 100644
> --- a/mysql-test/main/cte_nonrecursive.result
> +++ b/mysql-test/main/cte_nonrecursive.result
> @@ -244,7 +244,7 @@ with t as (select distinct a from t1 where b >= 'c')
>  select * from t as r1, t as r2 where r1.a=r2.a;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY   ALL NULLNULLNULLNULL8   
> Using where
> -1PRIMARY   ref key0key05   r1.a2   
> +1PRIMARY   ref key0key05   r1.a1   
>  3DERIVED t1  ALL NULLNULLNULLNULL8   Using 
> where; Using temporary
>  2DERIVED t1  ALL NULLNULLNULLNULL8   Using 
> where; Using temporary
>  explain
> @@ -253,7 +253,7 @@ select * from (select distinct a from t1 where b >= 'c') 
> as r1,
>  where r1.a=r2.a;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY   ALL NULLNULLNULLNULL8   
> Using where
> -1PRIMARY   ref key0key05   r1.a2   
> +1PRIMARY   ref key0key05   r1.a1   
>  3DERIVED t1  ALL NULLNULLNULLNULL8   Using 
> where; Using temporary
>  2DERIVED t1  ALL NULLNULLNULLNULL8   Using 
> where; Using temporary
>  # two references to t specified by a query
> @@ -369,7 +369,7 @@ select c as a from t2 where c < 4)
>  select * from t2,t where t2.c=t.a;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY t2  ALL NULLNULLNULLNULL4   Using 
> where
> -1PRIMARY   ref key0key05   test.t2.c   
> 2   
> +1PRIMARY   ref key0key05   test.t2.c   
> 1   
>  2DERIVED t1  ALL NULLNULLNULLNULL8   Using 
> where
>  3UNION   t2  ALL NULLNULLNULLNULL4   Using 
> where
>  NULL UNION RESULT  ALL NULLNULLNULLNULL
> NULL
> @@ -381,7 +381,7 @@ select c as a from t2 where c < 4) as t
>  where t2.c=t.a;
>  id   select_type table   typepossible_keys   key key_len ref 
> rowsExtra
>  1PRIMARY t2  ALL NULLNULLNULLNULL4   Using 
> where
> -1PRIMARY   ref key0key05   test.t2.c   
> 2   
> +1PRIMARY   ref key0key05   test.t2.c   
> 1   
>  2DERIVED t1  ALL NULLNULLNULLNULL8   Using 
> where
>  3UNION   t2  ALL NULLNULLNULLNULL4   Using 
> where
>  NULL UNION RESULT  ALL NULLNULLNULLNULL
> NULL
> diff --git a/mysql-test/main/derived.result b/mysql-test/main/derived.result
> index f0d0289c1ce..857246d68b4 100644
> --- a/mysql-test/main/derived.result
> +++ b/mysql-test/main/derived.result
> @@ -1195,3 +1195,57 @@ drop table t1,t2,t3;
>  #
>  # End of 10.2 tests
>  #
> +#
> +# MDEV-9959: A serious MariaDB server performance bug
> +#
> +create table t1(a int);
> +insert into t1 values (1),(2),(3),(4),(5),(6);
> +create table t2(a int, b int,c int);
> +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5);
> +create table t3(a int, b int);
> +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2);
> +table "" should have type=ref and rows=1
> +one select in derived table
> +with distinct
> +analyze select *

Re: [Maria-developers] [Commits] 88a80e92dc4: MDEV-9959: A serious MariaDB server performance bug

[Sergey Petrunia]

Hi Varun,

On Wed, Apr 24, 2019 at 01:31:38PM +0530, Varun wrote:
> revision-id: 88a80e92dc444ce30718f3e08d3ab66fb02bcea4 
> (mariadb-10.3.10-284-g88a80e92dc4)
> parent(s): 3032cd8e91f1e1ead8b6f941e75cd29e473e7eaa
> author: Varun Gupta
> committer: Varun Gupta
> timestamp: 2019-04-24 13:31:24 +0530
> message:
> 
> MDEV-9959: A serious MariaDB server performance bug
> 
> Step #2: If any field in the select list of the derived tables is present in 
> the group by list also , then we are again guaranteed
> that ref access to the derived table would always produce one row per key.
 
...

> diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc
> index c52005e7683..aa645ac00ca 100644
> --- a/sql/sql_lex.cc
> +++ b/sql/sql_lex.cc
> @@ -7617,6 +7617,68 @@ Item 
> *st_select_lex::build_cond_for_grouping_fields(THD *thd, Item *cond,
>  }
>  
>  
> +/**
> +  Check if any any item in the group by list is also present in the 
> select_list
> +  @retval true: All elements common between select and group by list
> +*/
> +
> +void st_select_lex::is_group_by_prefix(KEY *keyinfo)

I think the name of the function is poor - "is_something()" hints at hints at 
boolean return
value, and I would say it doesn't imply any side effects.

Something like 'mark_derived_index_stats()' would be better, I think.


> +{
> +  uint key_parts= keyinfo->usable_key_parts;
> +  KEY_PART_INFO *key_part_info= keyinfo->key_part;
> +  bool found= FALSE;
> +
> +  if (key_parts < group_list.elements)
> +return;
> +
> +  uint matched_fields=0, i, j;
> +  Item *item;
> +
> +  for (i= 0; i < key_parts; key_part_info++, i++)
> +  {
> +uint fld_idx= key_part_info->fieldnr - 1;
> +item= join->fields_list.elem(fld_idx);
> +for (ORDER *order= group_list.first; order; order= order->next)
> +{
> +  Item *ord_item= order->item[0]->real_item();
> +  Item_equal *item_equal= ord_item->get_item_equal();
> +
> +  if (item_equal)
> +  {
> +Item_equal_fields_iterator it(*item_equal);
> +Item *equal_item;
> +while ((equal_item= it++))
> +{
> +  if (equal_item->eq(item, 0))
> +  {
> +matched_fields++;
> +found= TRUE;
> +break;
> +  }
> +}
> +  }
> +  else
> +  {
> +if (item->eq(ord_item, 0))
> +{
> +  matched_fields++;
> +  found= TRUE;
> +}
> +  }
> +  if (found)
> +break;
> +}
> +
> +if (matched_fields == group_list.elements)
> +{
> +  for (j=matched_fields - 1; j < key_parts; j++)
> +keyinfo->rec_per_key[j]= 1;

This doesn't always work. An example:

create table t1 (a int, b int, c int, d int);
insert into t1 select a,a,a,a from one_k;

create table t2 as select * from t1;

# Note the "a, a AS b" in the select list. This is intentional.

explain select * 
from 
  t2,
  (select a, a as b from t1 group by a,b) TBL
where
  t2.a=TBL.a and t2.b=TBL.b;

Here, the select output is not guaranteed to be unique (as it contains only one
of two GROUP BY columns). But I see the execution to reac the above line,
assigning keyinfo->rec_per_key[j]= 1.

> +  return;
> +}
> +found= FALSE;
> +  }
> +}
> +
>  int set_statement_var_if_exists(THD *thd, const char *var_name,
>  size_t var_name_length, ulonglong value)
>  {

A general comment: do you think we should expose this in optimizer trace? 

At the moment, Derived-with-keys optimization doesn't print anything there
(neither in MariaDB nor in MySQL).  One can see the generated KEYUSEs when they
are printed.

But perhaps we could add a node that would print that a derived table was
added, with this cardinality, and index with these keypart cardinalities. 

This would make debugging easier. Any thoughts?

BR
 Sergei
-- 
Sergei Petrunia, Software Developer
MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog



___
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help   : https://help.launchpad.net/ListHelp