Re: next derby release

[Bryan Pendleton]

I, too, would like to see a Derby release prior to summer 2018.

I am comfortable with either proposal.

Thanks for volunteering to make this happen! I'll chip in as much as I can.

bryan

[jira] [Updated] (DERBY-6905) Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6905:
---
Attachment: dblook_tests.diff

I agree with Rick's assessment that these tests are failing due to the new 
column we added to SYSCOLUMNS.

And, I believe that the output is now correctly including the new column.

Attached 'dblook_tests.diff' is a patch which promotes the output files; with 
this patch applied, these tests pass for me.

Vardhan, can you please review the tests and this patch and let me know what 
you think?

> Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column
> ---
>
> Key: DERBY-6905
> URL: https://issues.apache.org/jira/browse/DERBY-6905
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 10.13.1.0
>Reporter: Bryan Pendleton
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Fix For: 10.14.0.0
>
> Attachments: dblook_tests.diff
>
>
> DERBY-6542 changed IDENTITY columns to build upon the underlying
> SEQUENCE objects. SEQUENCE objects can have a CYCLE option
> set for them, and as of DERBY-6852, the CYCLE option controls the
> behavior of the IDENTITY column that uses that SEQUENCE.
> It would be nice if SYS.SYSCOLUMNS would report the CYCLE option
> that is in effect for a IDENTITY column in that column's row in SYSCOLUMNS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DERBY-6905) Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053316#comment-16053316
 ] 

Bryan Pendleton commented on DERBY-6905:


The 'derbytools' suite only runs dblook_test and dblook_test_territory. The 
other two tests are part of 'derbynetclientmats'.

So in order to exercise dblook_test_net and dblook_test_net_territory, it is 
necessary to add a 5th step, in a different empty directory: 
5) java org.apache.derbyTesting.functionTests.harness.RunSuite 
derbynetclientmats


> Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column
> ---
>
> Key: DERBY-6905
> URL: https://issues.apache.org/jira/browse/DERBY-6905
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 10.13.1.0
>Reporter: Bryan Pendleton
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Fix For: 10.14.0.0
>
>
> DERBY-6542 changed IDENTITY columns to build upon the underlying
> SEQUENCE objects. SEQUENCE objects can have a CYCLE option
> set for them, and as of DERBY-6852, the CYCLE option controls the
> behavior of the IDENTITY column that uses that SEQUENCE.
> It would be nice if SYS.SYSCOLUMNS would report the CYCLE option
> that is in effect for a IDENTITY column in that column's row in SYSCOLUMNS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DERBY-6905) Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053305#comment-16053305
 ] 

Bryan Pendleton commented on DERBY-6905:


I reproduced (I think) the failures by doing the following:

1) building jars (ant buildJars)
2) creating a new empty directory and cd'ing there
3) export 
CLASSPATH=/path/to/derbyrun.jar:/path/to/derbyTesting.jar:/path/to/junit.jar
4) java org.apache.derbyTesting.functionTests.harness.RunSuite derbytools

The test results appear in 'derbytools_report.txt'.

Vardhan, can you try these steps in your environment and see if the test 
failures appear for you?

thanks,

bryan

> Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column
> ---
>
> Key: DERBY-6905
> URL: https://issues.apache.org/jira/browse/DERBY-6905
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 10.13.1.0
>Reporter: Bryan Pendleton
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Fix For: 10.14.0.0
>
>
> DERBY-6542 changed IDENTITY columns to build upon the underlying
> SEQUENCE objects. SEQUENCE objects can have a CYCLE option
> set for them, and as of DERBY-6852, the CYCLE option controls the
> behavior of the IDENTITY column that uses that SEQUENCE.
> It would be nice if SYS.SYSCOLUMNS would report the CYCLE option
> that is in effect for a IDENTITY column in that column's row in SYSCOLUMNS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: broken tests on trunk

[Bryan Pendleton]

I reproduced (I think) the failures by doing the following:

1) building jars (ant buildJars)
2) creating a new empty directory and cd'ing there
3) export
CLASSPATH=/path/to/derbyrun.jar:/path/to/derbyTesting.jar:/path/to/junit.jar
4) java org.apache.derbyTesting.functionTests.harness.RunSuite derbytools

The test results appear in 'derbytools_report.txt'.

thanks,

bryan

[jira] [Reopened] (DERBY-6905) Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton reopened DERBY-6905:


Reopening because there appear to be some regression tests failing after this 
change. See the following for more details:

http://mail-archives.apache.org/mod_mbox/db-derby-dev/201706.mbox/%3C728b613e-329b-8907-58ee-4cc38473a742%40gmail.com%3E

> Enhance SYS.SYSCOLUMNS to report the CYCLE value for an IDENTITY column
> ---
>
> Key: DERBY-6905
> URL: https://issues.apache.org/jira/browse/DERBY-6905
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 10.13.1.0
>Reporter: Bryan Pendleton
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Fix For: 10.14.0.0
>
>
> DERBY-6542 changed IDENTITY columns to build upon the underlying
> SEQUENCE objects. SEQUENCE objects can have a CYCLE option
> set for them, and as of DERBY-6852, the CYCLE option controls the
> behavior of the IDENTITY column that uses that SEQUENCE.
> It would be nice if SYS.SYSCOLUMNS would report the CYCLE option
> that is in effect for a IDENTITY column in that column's row in SYSCOLUMNS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: broken tests on trunk

[Bryan Pendleton]

Hi Rick, I'm sorry for the stupid question, but: can you remind me how to
run these tests? Are they run by 'ant junit-all' or by running
'ant
-Dderby.junit.testclass=org.apache.derbyTesting.functionTests.tests.tools._Suite
junit-single'?

Or do I have to run them some other way?

FWIW, running the tools suite using ant is currently reporting no errors
for me.

thanks,

bryan


On Sat, Jun 17, 2017 at 12:35 PM, Rick Hillegas 
wrote:

> In an empty workspace sync'd to the head of trunk as of this morning, the
> following tests fail for me:
>
>   dblook_test_net
>   dblook_test_net_territory
>   dblook_test
>   dblook_test_territory
>
> It looks to me as though the diffs are caused by the addition of a new
> SYSCOLUMNS_AUTOINCREMENTINCCYCLE column to SYS.SYSCOLUMNS.
>
> Thanks,
> -Rick
>
>

[jira] [Comment Edited] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053198#comment-16053198
 ] 

Harshvardhan Gupta edited comment on DERBY-6940 at 6/18/17 1:56 PM:


Attaching an initial patch which add the following statistics information - 

1) The minimum value of each column appearing in an index. 
2) The maximum value of each column appearing in an index.
3) The null counts for each column appearing in an index.

I am currently storing column level values for minValue and maxValue as 
DataValueDescriptor object. I am not entirely sure if this is the best way to 
store it for later use in comparison for determining selectivity estimates.

To invoke the procedure extraStatistics call 
SYSCS_UTIL.SYSCS_UPDATE_STATISTICS().
Currently the extraStatistics procedure is not invoked by the create Index 
statement, we will need to write a variant for "Create Index" statements to 
collect the extra statistics during index creation time as well.




was (Author: harshvardhan145):
Attaching an initial patch which add the following statistics information - 

1) The minimum value of each column appearing in an index. 
2) The maximum value of each column appearing in an index.
3) The null counts for each column appearing in an index.

I am currently storing column level values as DataValueDescriptor object. I am 
not entirely sure if this is the best way to store it for later use in 
comparison for determining selectivity estimates.

To invoke the procedure extraStatistics call 
SYSCS_UTIL.SYSCS_UPDATE_STATISTICS().
Currently the extraStatistics procedure is not invoked by the create Index 
statement, we will need to write a variant for "Create Index" statements to 
collect the extra statistics during index creation time as well.



> Enhance derby statistics for more accurate selectivity estimates.
> -
>
> Key: DERBY-6940
> URL: https://issues.apache.org/jira/browse/DERBY-6940
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Attachments: derby-6940.diff
>
>
> Derby should collect extra statistics during index build time, statistics 
> refresh time which will help optimizer make more precise selectivity 
> estimates and chose better execution paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053201#comment-16053201
 ] 

Harshvardhan Gupta edited comment on DERBY-6940 at 6/18/17 1:53 PM:


Additionally we need to work on upgrade procedures, while trying to access 
statistics info of an index created before this patch, Derby will throw a NPE 
due to absence of the new statistics in those indexes.


was (Author: harshvardhan145):
Additionally we need to work on upgrade procedures, while trying to access 
statistics info of an index created before this patch, Derby will throw a NPE 
due to absence of the new statistics. 

> Enhance derby statistics for more accurate selectivity estimates.
> -
>
> Key: DERBY-6940
> URL: https://issues.apache.org/jira/browse/DERBY-6940
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Attachments: derby-6940.diff
>
>
> Derby should collect extra statistics during index build time, statistics 
> refresh time which will help optimizer make more precise selectivity 
> estimates and chose better execution paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053201#comment-16053201
 ] 

Harshvardhan Gupta commented on DERBY-6940:
---

Additionally we need to work on upgrade procedures, while trying to access 
statistics info of an index created before this patch, Derby will throw a NPE 
due to absence of the new statistics. 

> Enhance derby statistics for more accurate selectivity estimates.
> -
>
> Key: DERBY-6940
> URL: https://issues.apache.org/jira/browse/DERBY-6940
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Attachments: derby-6940.diff
>
>
> Derby should collect extra statistics during index build time, statistics 
> refresh time which will help optimizer make more precise selectivity 
> estimates and chose better execution paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harshvardhan Gupta updated DERBY-6940:
--
Attachment: derby-6940.diff

Attaching an initial patch which add the following statistics information - 

1) The minimum value of each column appearing in an index. 
2) The maximum value of each column appearing in an index.
3) The null counts for each column appearing in an index.

I am currently storing column level values as DataValueDescriptor object. I am 
not entirely sure if this is the best way to store it for later use in 
comparison for determining selectivity estimates.

To invoke the procedure extraStatistics call 
SYSCS_UTIL.SYSCS_UPDATE_STATISTICS().
Currently the extraStatistics procedure is not invoked by the create Index 
statement, we will need to write a variant for "Create Index" statements to 
collect the extra statistics during index creation time as well.



> Enhance derby statistics for more accurate selectivity estimates.
> -
>
> Key: DERBY-6940
> URL: https://issues.apache.org/jira/browse/DERBY-6940
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
>Priority: Minor
> Attachments: derby-6940.diff
>
>
> Derby should collect extra statistics during index build time, statistics 
> refresh time which will help optimizer make more precise selectivity 
> estimates and chose better execution paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harshvardhan Gupta updated DERBY-6940:
--
Description: Derby should collect extra statistics during index build time, 
statistics refresh time which will help optimizer make more precise selectivity 
estimates and chose better execution paths.

> Enhance derby statistics for more accurate selectivity estimates.
> -
>
> Key: DERBY-6940
> URL: https://issues.apache.org/jira/browse/DERBY-6940
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
>Priority: Minor
>
> Derby should collect extra statistics during index build time, statistics 
> refresh time which will help optimizer make more precise selectivity 
> estimates and chose better execution paths.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DERBY-6938) Obtain cardinality estimates and true estimates for base tables as well as for intermediate results for queries involving multiple joins.

[Harshvardhan Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053197#comment-16053197
 ] 

Harshvardhan Gupta commented on DERBY-6938:
---

Tracking work on statistics enhancement at 
https://issues.apache.org/jira/browse/DERBY-6940

>  Obtain cardinality estimates and true estimates for base tables as well as 
> for intermediate results for queries involving multiple joins. 
> ---
>
> Key: DERBY-6938
> URL: https://issues.apache.org/jira/browse/DERBY-6938
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
> Attachments: explain.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

Harshvardhan Gupta created DERBY-6940:
-

 Summary: Enhance derby statistics for more accurate selectivity 
estimates.
 Key: DERBY-6940
 URL: https://issues.apache.org/jira/browse/DERBY-6940
 Project: Derby
  Issue Type: Sub-task
Reporter: Harshvardhan Gupta
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DERBY-6940) Enhance derby statistics for more accurate selectivity estimates.

[Harshvardhan Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harshvardhan Gupta reassigned DERBY-6940:
-

Assignee: Harshvardhan Gupta

> Enhance derby statistics for more accurate selectivity estimates.
> -
>
> Key: DERBY-6940
> URL: https://issues.apache.org/jira/browse/DERBY-6940
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Harshvardhan Gupta
>Assignee: Harshvardhan Gupta
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)