[Firebird-devel] [FB-Tracker] Created: (CORE-5127) GBAK should be able to make a backup of firebird files with previus ODS.

[Anderson Franco (JIRA)]

GBAK should be able to make a backup of firebird files with previus ODS.  
--

 Key: CORE-5127
 URL: http://tracker.firebirdsql.org/browse/CORE-5127
 Project: Firebird Core
  Issue Type: Improvement
  Components: GBAK
Affects Versions: 3.0 RC2
 Environment: Windows 7, 8, 10, Server
Reporter: Anderson Franco
Priority: Minor


I know I can backup/uninstall old FB/install new FB3/restore.

I think It would be easier to migrate from previuos versions of FB if FB3 could 
backup databases with previous ODS. The goal is uninstall older Firebird, 
install newer, backup and immediatly restore a database. This way there is no 
need of two versions of firebird installed. 

It would be also useful when someone send you a database and you are running 
only FB3.

Thank you.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: 
http://tracker.firebirdsql.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Firebird-devel] [FB-Tracker] Created: (CORE-5126) Installation of Firebird 3.0 beta 2 totally ignores my sysdba login information

[Danniel Corbit (JIRA)]

Installation of Firebird 3.0 beta 2 totally ignores my sysdba login information
---

 Key: CORE-5126
 URL: http://tracker.firebirdsql.org/browse/CORE-5126
 Project: Firebird Core
  Issue Type: Bug
  Components: Security
Affects Versions: 3.0 Beta 2
 Environment: Windows Server 2012 R2 64 bit
Reporter: Danniel Corbit
Priority: Blocker


Installation of Firebird offers a screen to set the system DBA username and 
password at install time.
It allowed me to fill out the form, but totally ignored the results.

You cannot change the system owner (for instance) from SYSDBA to SA
You cannot change the system owner password.  No matter what you type (and it 
does get accepted!) the sysdba password is masterkey when you are done.
This is a security breach so severe that the database is unusable.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: 
http://tracker.firebirdsql.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Jim Starkey]

On 3/3/2016 2:30 PM, les...@lsces.co.uk wrote:
Initially when I ran websites, the normal practice was to save 
everything in the database, pictures, pdf's, documents, and so on. The 
databases soon got too large and while just having one file to back up 
was nice, NOT using blobs for that content was a lot easier. Rsync to 
backup machine, including a nightly backup of the 'real' data. The 
cross over point is perhaps when you need to do a text search, but 
even that is easier if a pdf or doc is preprocessed to provide a copy 
as simple text. Which nowadays is what ends up in my blobs. Replace 
vast blob zone with the native file system?


Where i can see a useful segregation is archival data which will never 
be modified. Would be very usefull if all that could be backed up on a 
Slow cycle, and only the real dynamic data kept in the primary table 
space? But one can do most of that by having multiple databases anyway?


You know, that's a really good idea.  Disk is almost free, too cheap to 
meter (so to speak).  Store something once and just don't worry if it 
goes out of style...




Sent from my android device so quoting is crap ... need to kill these 
painful email clients!


-Original Message-
From: Ann Harrison 
To: For discussion among Firebird Developers 


Sent: Thu, 03 Mar 2016 18:37
Subject: Re: [Firebird-devel] RFC: Tablespaces

On Thu, Mar 3, 2016 at 1:02 PM, Jim Starkey > wrote:



>> Non-goals:
>>
...
>>   2. Store blobs in other than the table's data space.
> Why not allow blobs to be separated from regular data ?

OK, reasonable question.  Obviously they could, but it would require
either storing small blobs off page or changing the mechanism used for
blob ids, or both.  It also runs the risk of the records and blobs
diverging, which is very, very bad.

I think the benefit of separating records and blobs is quite limited.
Large blobs have at least their rear end stored on blob pages that
aren't scanned for exhaustive retrievals.  Moving them to a separate
data space within the same table space so they don't share the record
number space with records is well worth considering.


I think the problem is with the size of backups and the amount of time
taken by a backup/restore for a database with a significant number of
large blobs.

Cheers,

Ann


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140


Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Slavomir Skopalik]

1. Data can (already did) at any point, not only at specific, that you 
marks by '!'.
2. If packed record doesn't fit into one page, control sequence is 
parsed from end.
3. When rest of sequence will fit, compression is started again from 
beginning.
4. If fragment has odd length, zero is added.

if you will generate compressed buffer instead control sequence, you can use
any compress/encoding schema.
If you will store this compressed buffer in one or more fragments is it 
irrelevant.
You can join/splitting as you want. But for decompression  you have to 
collect all fragments into buffer.

Slavek

> 03.03.2016 18:34, Slavomir Skopalik wrote:
>> But value encoding cannot be implemented until we switch from fragment
>> compression to true record level compression.
> I was not speaking about any encoding, just about compacting the record.
>
> Fragments are not compressed independently. The whole record is being
> prepared for compression (control bytes are generated) and then splitted
> into multiple chunks accordingly to the control bytes, so that a single
> compressed sequence is not interrupted in the middle:
>
> 'oooiiiuee'
> {-4, 'a'} ! {-3, 'o'} ! {-3, 'i'} ! {1, 'u'} ! {2, 'ee'}
>
> This compressed sequence can be fragmented at any of the "!" points.
> This is needed to decompress them separately, without copying all the
> fragments into a single buffer before decompressing, i.e. fragment by
> fragment.
>
> But for the pack/unpack approach, we just copy data in chunks. Some
> field may have first N bytes stored in fragment 1 and the remaining M
> bytes stored in fragment 2. We know we need N+M bytes for the record. So
> we copy N bytes from fragment 1, release the page, fetch fragment 2 and
> continue copying N+1..M bytes into the record. Fragmenting is even
> easier than now, it can happen at any place.
>
> Am I missing something obvious?
>



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Alexandre Benson Smith]

Em 3/3/2016 04:04, Dmitry Yemanov escreveu:
> 03.03.2016 09:48, Dmitry Yemanov wrote:
>>> Blobs could be moved into separate tablespace. It could make backup of
>>> "data" tablespace faster and smaller. We can even think about "offline"
>>> tablespace.
>> Really good idea. Maybe more important for an average FB user than the
>> other usage cases.
> BTW, if blob is declared as belonging to some tablespace, should level-0
> blobs be also moved to its own tablespace or stored in the table tablespace?
>
> And another question: does RDB$PAGES need to include the pagespace ID or
> we never allow tablespaces for TIP, generator pages and system tables?
> Pagespace ID for PP/IRP can be retrieved from RDB$ tables based on
> relation_id, so I don't see a problem to store just the "local" page number.
>
>
> Dmitry

I think all the blobs should go to the blob tablespace, it's a common 
approach to create a 1:1 table just to store the blobs, preventing it to 
be stored inside the data pages.

Any way, it could be an option when the users define the table.

see you !




--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Slavomir Skopalik]

Hi Jim,
Elekt Labs RLE is ready to use and is better then current one.

The best one have to be developed and I will help if I can.

The question is still same: Is it right time or will be postponed to V5 
(2024+) ?

Slavek

>
> Excuse me, but faster than the current one is not the question. The 
> question is what is the best one.



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Adriano dos Santos Fernandes]

Em 03/03/2016 13:43, Vlad Khorsun escreveu:

> 
>I would add
> 
> 4. Allow to specify FW and Read-Only settings on per tablespace level
> 5. Ability to bring individual tablespace off-line (and back online, of 
> course)
> 

...

It's not a tablespace feature, but a schema feature integrated with
tablespaces, each schema should have a default tablespace.


Adriano

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[lester]

Initially when I ran websites, the normal practice was to save everything in 
the database, pictures, pdf's, documents, and so on. The databases soon got too 
large and while just having one file to back up was nice, NOT using blobs for 
that content was a lot easier. Rsync to backup machine, including a nightly 
backup of the 'real' data. The cross over point is perhaps when you need to do 
a text search, but even that is easier if a pdf or doc is preprocessed to 
provide a copy as simple text. Which nowadays is what ends up in my blobs. 
Replace vast blob zone with the native file system?

Where i can see a useful segregation is archival data which will never be 
modified. Would be very usefull if all that could be backed up on a Slow cycle, 
and only the real dynamic data kept in the primary table space? But one can do 
most of that by having multiple databases anyway?

Sent from my android device so quoting is crap ... need to kill these painful 
email clients!

-Original Message-
From: Ann Harrison 
To: For discussion among Firebird Developers 

Sent: Thu, 03 Mar 2016 18:37
Subject: Re: [Firebird-devel] RFC: Tablespaces

On Thu, Mar 3, 2016 at 1:02 PM, Jim Starkey  wrote:

>
> >> Non-goals:
> >>
> ...
> >>   2. Store blobs in other than the table's data space.
> > Why not allow blobs to be separated from regular data ?
>
> OK, reasonable question.  Obviously they could, but it would require
> either storing small blobs off page or changing the mechanism used for
> blob ids, or both.  It also runs the risk of the records and blobs
> diverging, which is very, very bad.
>
> I think the benefit of separating records and blobs is quite limited.
> Large blobs have at least their rear end stored on blob pages that
> aren't scanned for exhaustive retrievals.  Moving them to a separate
> data space within the same table space so they don't share the record
> number space with records is well worth considering.
>

I think the problem is with the size of backups and the amount of time
taken by a backup/restore for a database with a significant number of
large blobs.

Cheers,

Ann
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Ann Harrison]

On Thu, Mar 3, 2016 at 1:02 PM, Jim Starkey  wrote:

>
> >> Non-goals:
> >>
> ...
> >>   2. Store blobs in other than the table's data space.
> > Why not allow blobs to be separated from regular data ?
>
> OK, reasonable question.  Obviously they could, but it would require
> either storing small blobs off page or changing the mechanism used for
> blob ids, or both.  It also runs the risk of the records and blobs
> diverging, which is very, very bad.
>
> I think the benefit of separating records and blobs is quite limited.
> Large blobs have at least their rear end stored on blob pages that
> aren't scanned for exhaustive retrievals.  Moving them to a separate
> data space within the same table space so they don't share the record
> number space with records is well worth considering.
>

I think the problem is with the size of backups and the amount of time
taken by a backup/restore for a database with a significant number of
large blobs.

Cheers,

Ann
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Jim Starkey]

On 3/3/2016 11:43 AM, Vlad Khorsun wrote:
> 03.03.2016 17:57, Jim Starkey wrote:
>> The place to start is with a clear, well understood, and agreed set of 
>> requirements.  Here is a suggested starting point.
>>
>> First, some definitions.  A "data space" is the collection of pointer and 
>> data pages that hold the data from a single table.  An
>> "index" (in this context), is the set of index pages that compromise a 
>> single index on a table.  A "database object" is either a
>> data space or an index.  A "table space" consists of a page structured file, 
>> a collection of management pages, and a page number
>> space that can hold zero or more database objects.
>>
>> The minimal requirements:
>>
>>   1. To extend the theoretical number of database pages in a database beyond 
>> 2^32.
>>   2. To be as inobstrusive as possible so that no explicit configuration is 
>> required for databases that don't use table spaces.  In
>>  other words, a default (root) table space.
>>   3. To allow a user to define and manage additional table spaces.
>>   4. To allow a user to define tables spaces for individual tables and 
>> individual indexes.  In other words, while indexes may default
>>  to the same table space of the table data space, indexes may reside in 
>> different table spaces.
>>   5. A user should be able to move a table data space or index from one 
>> table space to another.  Whether this is an on-line or
>>  off-line operation is deferred to specific proposals.
>>   6. It should be possible to bring up a database with some or all table 
>> spaces unavailable other than the root.
>>   7. A table space should have sufficient internal integrity to tolerate 
>> operational problems resulting improper management of
>>  database files.
>>   8. It must be necessary to allow the filename of an offline table space to 
>> be changed.
>Basically, i agree with above.
>
>> Personally, I think the following are also important, but not requirements:
>>
>>   1. There should be minimal changes to the on-disk structure. More 
>> specifically, pages references in the ODS should remain as 32
>>  bits.  This precludes allowing a database object to straggle table 
>> spaces.
>>   2. There should be as few changes as possible to internal page, index, and 
>> data handling mechanism other than tracking the table
>>  space for database objects for the CCH interface.
>>   3. It should be anticipated that table spaces will be abused, for example 
>> replacing table space files with older versions. To the
>>  degree feasible, this should be supported, and where not supported, 
>> detected.
> I would add
>
> 4. Allow to specify FW and Read-Only settings on per tablespace level
> 5. Ability to bring individual tablespace off-line (and back online, of 
> course)

#5 is a candidate for promotion to a requirement, but keep in mind that 
some mechanism must be invented to sweep any off-line table spaces 
before they come on-line lest record versions created by abandoned or 
rolled back transactions suddenly re-appear.  This should be reasonably 
straightforward as the TIPs have retain the old states.

>
>> Non-goals:
>>
>>   1. Change the size of record numbers
>>   2. Store blobs in other than the table's data space.
> Why not allow blobs to be separated from regular data ?

OK, reasonable question.  Obviously they could, but it would require 
either storing small blobs off page or changing the mechanism used for 
blob ids, or both.  It also runs the risk of the records and blobs 
diverging, which is very, very bad.

I think the benefit of separating records and blobs is quite limited.  
Large blobs have at least their rear end stored on blob pages that 
aren't scanned for exhaustive retrievals.  Moving them to a separate 
data space within the same table space so they don't share the record 
number space with records is well worth considering.

>
>>   3. Implement any ODS change other than those required to support table 
>> spaces (other projects are fine, but should be layered on,
>>  not part of, table spaces).


> Regards,
> Vlad
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
> Firebird-Devel mailing list, web interface at 
> https://lists.sourceforge.net/lists/listinfo/firebird-devel


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user 

Re: [Firebird-devel] RFC: Tablespaces

[Ann Harrison]

On Wed, Mar 2, 2016 at 4:51 PM, Vlad Khorsun 
wrote:

>
>Blobs could be moved into separate tablespace. It could make backup of
> "data" tablespace faster and smaller. We can even think about "offline"
> tablespace.
>
>
Interesting idea. I'm not quite sure how it would work ... new blobs should
be
backed up, I'd guess, and old ones ignored.  That's pretty far from the way
gbak
works.  But gbak may not be the right tool for terabyte databases.

 Cheers,

Ann
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Ann Harrison]

On Wed, Mar 2, 2016 at 4:23 PM, Dmitry Yemanov  wrote:

>

> When we speak about tablespaces, it usually means that the database
> consists of multiple files and different database object are stored in
> different files. Each such file is named within a database and called a
> tablespace.
>

It's probably not a reasonable concern at this point, but applying Oracle
tuning mechanisms to a Firebird database probably won't have the
performance benefits users expect.  It's going to take a lot of education to
convince people to use a feature they think they understand in a way that
work with Firebird,  Maybe I'm just arguing for a different name.

Cheers,

Ann
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[James Starkey]

On Thursday, March 3, 2016, Dmitry Yemanov  wrote:

> 03.03.2016 19:16, Jim Starkey wrote:
>
> > On 3/3/2016 11:10 AM, Dmitry Yemanov wrote:
> >> Am I missing something obvious?
> >
> > Yeah.  Data.
>
> I was not talking about value-based encoding (which surely requires a
> different approach).
>
>
>
And why not?  Are you so married to 35 year old technology designed to run
on platforms with less computing power than contemporary parking meters
that you can't envision that times have changed and there are new ideas in
the world?

Come on, Dmitry, you're a bright lad.  Consider the problem in depth and
let your mind range over the many possible solutions.  You can do it.  It
only hurts a little and only at first.


Dmitry
>
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
>


-- 
Jim Starkey
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Vlad Khorsun]

03.03.2016 17:57, Jim Starkey wrote:
> The place to start is with a clear, well understood, and agreed set of 
> requirements.  Here is a suggested starting point.
>
> First, some definitions.  A "data space" is the collection of pointer and 
> data pages that hold the data from a single table.  An
> "index" (in this context), is the set of index pages that compromise a single 
> index on a table.  A "database object" is either a
> data space or an index.  A "table space" consists of a page structured file, 
> a collection of management pages, and a page number
> space that can hold zero or more database objects.
>
> The minimal requirements:
>
>  1. To extend the theoretical number of database pages in a database beyond 
> 2^32.
>  2. To be as inobstrusive as possible so that no explicit configuration is 
> required for databases that don't use table spaces.  In
> other words, a default (root) table space.
>  3. To allow a user to define and manage additional table spaces.
>  4. To allow a user to define tables spaces for individual tables and 
> individual indexes.  In other words, while indexes may default
> to the same table space of the table data space, indexes may reside in 
> different table spaces.
>  5. A user should be able to move a table data space or index from one table 
> space to another.  Whether this is an on-line or
> off-line operation is deferred to specific proposals.
>  6. It should be possible to bring up a database with some or all table 
> spaces unavailable other than the root.
>  7. A table space should have sufficient internal integrity to tolerate 
> operational problems resulting improper management of
> database files.
>  8. It must be necessary to allow the filename of an offline table space to 
> be changed.

  Basically, i agree with above.

> Personally, I think the following are also important, but not requirements:
>
>  1. There should be minimal changes to the on-disk structure. More 
> specifically, pages references in the ODS should remain as 32
> bits.  This precludes allowing a database object to straggle table spaces.
>  2. There should be as few changes as possible to internal page, index, and 
> data handling mechanism other than tracking the table
> space for database objects for the CCH interface.
>  3. It should be anticipated that table spaces will be abused, for example 
> replacing table space files with older versions. To the
> degree feasible, this should be supported, and where not supported, 
> detected.

   I would add

4. Allow to specify FW and Read-Only settings on per tablespace level
5. Ability to bring individual tablespace off-line (and back online, of course)

> Non-goals:
>
>  1. Change the size of record numbers
>  2. Store blobs in other than the table's data space.

   Why not allow blobs to be separated from regular data ?

>  3. Implement any ODS change other than those required to support table 
> spaces (other projects are fine, but should be layered on,
> not part of, table spaces).

Regards,
Vlad


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 17:38, Dmitry Yemanov wrote:
> Pretty no difference for short hash strings, thanks.

   Your conslusion is...?

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 17:25, Vlad Khorsun wrote:
>>  Tests shown no drawback for longer data.
> Does you test the code without that two if's ?

   Yes, I specially compared performance with and without them. Function with 
these ifs 
won on short (1-2 bytes) data and shown the same performance (within timer 
imprecision) on 
longer samples. If you are getting different result - public it, please.

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dmitry Yemanov]

03.03.2016 19:33, Dimitry Sibiryakov wrote:

> Original:
>> Elapsed time= 1.243 sec
vs
> CRC32:
>> Elapsed time= 1.238 sec

Pretty no difference for short hash strings, thanks.


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Dmitry Yemanov]

03.03.2016 19:16, Jim Starkey wrote:

> On 3/3/2016 11:10 AM, Dmitry Yemanov wrote:
>> Am I missing something obvious?
>
> Yeah.  Data.

I was not talking about value-based encoding (which surely requires a 
different approach).


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 17:13, Dmitry Yemanov wrote:
> Can you perform the same test but with INT join key?

   I made number of records in table i2 four times bigger to get rid of timer's 
jitter, 
but still...

Original:
> SQL> select count(*) from i1 join i2 on i1.a=i2.a;
>
> PLAN HASH (I2 NATURAL, I1 NATURAL)
>
> COUNT
> =
>72
>
> Current memory = 342876528
> Delta memory = 0
> Max memory = 342959008
> Elapsed time= 1.243 sec
> Buffers = 4
> Reads = 0
> Writes = 0
> Fetches = 659455

CRC32:

> SQL> select count(*) from i1 join i2 on i1.a=i2.a;
>
> PLAN HASH (I2 NATURAL, I1 NATURAL)
>
> COUNT
> =
>72
>
> Current memory = 354130256
> Delta memory = 0
> Max memory = 354217800
> Elapsed time= 1.238 sec
> Buffers = 4
> Reads = 0
> Writes = 0
> Fetches = 659455

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Vlad Khorsun]

03.03.2016 16:15, Dimitry Sibiryakov wrote:
> 03.03.2016 15:07, Vlad Khorsun wrote:
  Are you sure that lock manager doesn't work with data of one or two 
 bytes long? At
 these sizes the function had a weak spot without these ifs.
>>  Perfect. All other 99% cases (with key >= 4 bytes) must pay for nothing.
>
> Tests shown no drawback for longer data.

   Does you test the code without that two if's ?

Vlad


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Jim Starkey]

On 3/3/2016 11:10 AM, Dmitry Yemanov wrote:
> Am I missing something obvious? 

Yeah.  Data.

> Dmitry 
> --
>  
> Site24x7 APM Insight: Get Deep Visibility into Application Performance 
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month 
> Monitor end-to-end web transactions and take corrective actions now 
> Troubleshoot faster and improve end-user experience. Signup Now! 
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140 
> Firebird-Devel mailing list, web interface at 
> https://lists.sourceforge.net/lists/listinfo/firebird-devel 


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dmitry Yemanov]

03.03.2016 18:36, Dimitry Sibiryakov wrote:

> varchar(2000) filled with rpad('', 2000, uuid_to_char(gen_uuid())).

Can you perform the same test but with INT join key?


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Dmitry Yemanov]

03.03.2016 18:34, Slavomir Skopalik wrote:
>
> But value encoding cannot be implemented until we switch from fragment
> compression to true record level compression.

I was not speaking about any encoding, just about compacting the record.

Fragments are not compressed independently. The whole record is being 
prepared for compression (control bytes are generated) and then splitted 
into multiple chunks accordingly to the control bytes, so that a single 
compressed sequence is not interrupted in the middle:

'oooiiiuee'
{-4, 'a'} ! {-3, 'o'} ! {-3, 'i'} ! {1, 'u'} ! {2, 'ee'}

This compressed sequence can be fragmented at any of the "!" points. 
This is needed to decompress them separately, without copying all the 
fragments into a single buffer before decompressing, i.e. fragment by 
fragment.

But for the pack/unpack approach, we just copy data in chunks. Some 
field may have first N bytes stored in fragment 1 and the remaining M 
bytes stored in fragment 2. We know we need N+M bytes for the record. So 
we copy N bytes from fragment 1, release the page, fetch fragment 2 and 
continue copying N+1..M bytes into the record. Fragmenting is even 
easier than now, it can happen at any place.

Am I missing something obvious?


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dmitry Yemanov]

03.03.2016 18:49, Dimitry Sibiryakov wrote:

>> Out of curiosity, why do we need a separate crc32c.cpp?
>
> To let engine work on processors without SSE4.2 command set.

Ah, I missed processor-specific compilation flags for that module. 
Thanks for clarifying.


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 16:31, Dmitry Yemanov wrote:
> I had the same doubts about the first two if's, but
> Vlad already asked about them.

   And as I already said: they improve performance with short data without 
visible affect 
for long data. At least two months old tests showed me that. If anyone's tests 
proof 
contrary, these lines can be removed in no time.

> Out of curiosity, why do we need a separate crc32c.cpp?

   To let engine work on processors without SSE4.2 command set.

> Cannot the CRC32C function be defined inside Hash.cpp?

   It can. But whole file then will have to be compiled with SSE4.2 instruction 
set 
enabled. If (by chance) a compiler generate any SSE4 instruction anywhere 
outside of 
CRC32C() function, engine will crash on old processors and you'll spend a lot 
of time 
trying to find out why. (At least I did.)

> I hate to see such a conditional code:

   Yes, it is ugly, but it is the price for hardware acceleration for 
multiplatform projects.

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 16:20, Vlad Khorsun wrote:
>> will you believe my numbers?..
> Why not ?

   Exactly because of

> special case, but very far from real life, i'm afraid.

> I guess you have join on long strings.

   Not much long: varchar(2000) filled with rpad('', 2000, 
uuid_to_char(gen_uuid())).

 > T1 (128)
 > Primary pointer page: 180, Index root page: 181
 > Total formats: 1, used formats: 1
 > Average record length: 2021.06, total records: 32
 > Average version length: 0.00, total versions: 0, max versions: 0
 > Average fragment length: 0.00, total fragments: 0, max fragments: 0
 > Average unpacked length: 2006.00, compression ratio: 0.99
 > Pointer pages: 1, data page slots: 16
 > Data pages: 16, average fill: 50%
 > Primary pages: 16, secondary pages: 0, swept pages: 0
 > Empty pages: 5, full pages: 10
 > Fill distribution:
 >   0 - 19% = 5
 >  20 - 39% = 0
 >  40 - 59% = 1
 >  60 - 79% = 10
 >  80 - 99% = 0
 >
 > T2 (129)
 > Primary pointer page: 200, Index root page: 201
 > Total formats: 1, used formats: 1
 > Average record length: 2020.43, total records: 81920
 > Average version length: 0.00, total versions: 0, max versions: 0
 > Average fragment length: 0.00, total fragments: 0, max fragments: 0
 > Average unpacked length: 2006.00, compression ratio: 0.99
 > Pointer pages: 17, data page slots: 27176
 > Data pages: 27176, average fill: 75%
 > Primary pages: 27176, secondary pages: 0, swept pages: 0
 > Empty pages: 4, full pages: 27171
 > Fill distribution:
 >   0 - 19% = 4
 >  20 - 39% = 0
 >  40 - 59% = 1
 >  60 - 79% = 26766
 >  80 - 99% = 405

   I wonder why number of fetches is so high?..

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dmitry Yemanov]

03.03.2016 14:07, Dimitry Sibiryakov wrote:

> Ok, here you have it, attached.

Much better, thanks. I had the same doubts about the first two if's, but 
Vlad already asked about them.

Out of curiosity, why do we need a separate crc32c.cpp? Cannot the 
CRC32C function be defined inside Hash.cpp? I hate to see such a 
conditional code:

#if defined(_M_IX86) || defined(_M_X64) || defined(__x86_64__) || 
defined(__i386__)

duplicated across multiple files.


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Vlad Khorsun]

03.03.2016 16:51, Dimitry Sibiryakov wrote:
> 03.03.2016 15:07, Vlad Khorsun wrote:
>>>   I wouldn't expect much from lock manager, but hash joins can get 
>>> significant boost,
 especially on long keys.
>>  You don't need "server with dozen of cores" to check it, isn't is ?
>
> You are right, this part I can test even with my notebook. But will you 
> believe my
> numbers?..

   Why not ?

> Here they are:
>
>> SQL> show version;
>> ISQL Version: WI-V3.0.0.32342 Firebird 3.0 Release Candidate 2
>> Server version:
>> Firebird/Windows/Intel/i386 (access method), version "WI-V3.0.0.32342 
>> Firebird 3.0 Release Candidate 2"
>> on disk structure version 12.0
>> SQL> select count(*) from t1 join t2 on t1.a=t2.a;
>>
>> PLAN HASH (T2 NATURAL, T1 NATURAL)
>>
>>  COUNT
>> =
>> 32
>>
>> Current memory = 342822008
>> Delta memory = 0
>> Max memory = 342998536
>> Elapsed time= 1.775 sec
>> Buffers = 4
>> Reads = 0
>> Writes = 0
>> Fetches = 218290
>>
>> SQL> show version;
>> ISQL Version: WI-T4.0.0.32373 Firebird 4.0 Unstable
>> Server version:
>> Firebird/Windows/AMD/Intel/x64 (access method), version "WI-T4.0.0.32373 
>> Firebird 4.0 Unstable"
>> on disk structure version 12.0
>> SQL> select count(*) from t1 join t2 on t1.a=t2.a;
>>
>> PLAN HASH (T2 NATURAL, T1 NATURAL)
>>
>>  COUNT
>> =
>> 32
>>
>> Current memory = 354135072
>> Delta memory = 0
>> Max memory = 354316480
>> Elapsed time= 0.924 sec
>> Buffers = 4
>> Reads = 0
>> Writes = 0
>> Fetches = 218290

   Looks nice, but you "forget" to show table's DDL and how data was generated. 
I guess you have
join on long strings. It is ok as special case, but very far from real life, 
i'm afraid.

Vlad

PS I'm not against you or your code, i'm against your public behavior.

PPS Don't forget to test without first two if's :)

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[liviuslivius]

Vlad proposition is better then writting somethink on every transaction and 
should work without ovehead

look at this (if i understand Vlad's proposition correctly)

two fields
MAIN_DB_GUID, MAIN_OPENED
and the same for all table space files (as S_DB_GUID, S_OPENED)

1. First connect to db
2. Read DB_GUID and MAIN_OPENED
3. check if MAIN_OPENED = false
a) False = ok
b) True = Error
4. write back MAIN_OPENED = true
5. access to all table spaces in loop. read S_DB_GUID, S_OPENED
6. check if S_OPENED = false
a) False = ok
b) True = Error
7. compare MAIN_DB_GUID and S_DB_GUID
a) same=ok
b) diff=error
8. start closing database files
9. generete new MAIN_DB_GUID
10. write MAIN_DB_GUID back to db and MAIN_OPENED=false
11. close all table spaces
12. write replace S_DB_GUID with MAIN_DB_GUID  and S_OPENED=false

regards,
Karol Bieniaszewski


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 15:07, Vlad Khorsun wrote:
>>  I wouldn't expect much from lock manager, but hash joins can get 
>> significant boost,
>> >especially on long keys.
> You don't need "server with dozen of cores" to check it, isn't is ?

   You are right, this part I can test even with my notebook. But will you 
believe my 
numbers?..

   Here they are:

> SQL> show version;
> ISQL Version: WI-V3.0.0.32342 Firebird 3.0 Release Candidate 2
> Server version:
> Firebird/Windows/Intel/i386 (access method), version "WI-V3.0.0.32342 
> Firebird 3.0 Release Candidate 2"
> on disk structure version 12.0
> SQL> select count(*) from t1 join t2 on t1.a=t2.a;
>
> PLAN HASH (T2 NATURAL, T1 NATURAL)
>
> COUNT
> =
>32
>
> Current memory = 342822008
> Delta memory = 0
> Max memory = 342998536
> Elapsed time= 1.775 sec
> Buffers = 4
> Reads = 0
> Writes = 0
> Fetches = 218290
>
> SQL> show version;
> ISQL Version: WI-T4.0.0.32373 Firebird 4.0 Unstable
> Server version:
> Firebird/Windows/AMD/Intel/x64 (access method), version "WI-T4.0.0.32373 
> Firebird 4.0 Unstable"
> on disk structure version 12.0
> SQL> select count(*) from t1 join t2 on t1.a=t2.a;
>
> PLAN HASH (T2 NATURAL, T1 NATURAL)
>
> COUNT
> =
>32
>
> Current memory = 354135072
> Delta memory = 0
> Max memory = 354316480
> Elapsed time= 0.924 sec
> Buffers = 4
> Reads = 0
> Writes = 0
> Fetches = 218290

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Dmitry Yemanov]

All,

This is how I see the things.

RLE is good for record compression due to the following facts:

1) In-memory record has all the fields expanded up to their maximum 
size. Shorter values are padded with zeroes (that are easy to compress).

2) In-memory record stores the NULL bitmap and at the same time zaps the 
space of the NULL values with zeroes (that are easy to compress).

3) In-memory record has all the fields aligned to their respective 
boundaries. Unused gaps are also zapped with zeroes (that are easy to 
compress).

Of course, it also works good for small numbers stored in INT/BIGINT 
fields and for repeating character sequences in CHAR/VARCHAR fields. But 
I'd say that 95% of the RLE effect is covered by the aforementioned 
three cases, especially (1) and (2). Longish VARCHAR fields in UTF8 are 
an extreme example.

The legacy algorithm compresses up to 128 bytes into 2 bytes, i.e. has 
maximum compression ratio of 64x. It means that 10KB compresses into 156 
bytes while theoretically it could be compressed into three or four 
bytes. Quite a huge difference. This is what the suggested new algorithm 
(actually, a clever modification of the old one) is expected to solve. 
So far so good.

My question is would the RLE compression still useful if points (1) - 
(3) disappear. For example, the record is packed the following way:

- NULL bitmap
- alignment padding is skipped
- NULL fields are skipped
- VARCHARs are stored in their actual length
- probably the same for CHARs, just need to spend some CPU cycles to 
calculate the actual length (*)

In other words, records are packed/unpacked semantically, using their 
formats.

How such an approach compare to the old RLE algorithm? To the proposed 
one? Anyone willing to test?

This way, RLE could still be used for long VARCHARs, optionally. Or 
replaced with a value-based encoding or something else, again optionally.

I'd go even further and eliminate point (1) completely:

- record consists of two parts: fixed (described by the format) and 
variable (follows the fixed one)
- all fixed-size fields are stored as now
- VARCHAR is stored as {length, offset}
- offset points to its data stored in the tail

i.e. switch to variable-length records. It would cost us some extra 
reallocations at runtime, but not so many as one could think. It could 
dramatically reduce memory usage in some subsystems. And 
much-shorter-than-declared strings don't need mandatory RLE compression.

This surely still needs more thinking, but you get the idea.

(*) I doubt any sane person uses CHARs for strings longer than a few 
dozens of characters, so I wouldn't care much about optimizing their 
storage.


Dmitry

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Dalton Calford]

We use Oracle and table spaces.   We specify which drives/drive channels
hold which data/tables/indexes to optimize various OLAP/OLTP needs.

We also use firebird.   We use multiple files as large multi-raid drive
arrays have problems in that the larger arrays may have multiple drives
fail faster than new drives can rebuild the raid stripe.Raid six has
allowed our acceptable limit to increase, but, it does not alter the issue
that some database systems generate far more transactional data than what
most people can conceive of.   Although drive capacity is growing, the data
i/o channels that have to transfer that data have not been growing at the
same pace.   When you need to suddenly generate a new drive on a
RAID5/6/50/60 system when all the drives where purchased at around the same
time/batch, you can easily see three or four drives fail on the same system
within the same time period.  With smaller raids, the rebuild is well
within the safety zone of having the hot spares finish before another drive
starts to fail.  So there is a maximum drive safety size (which is a factor
of the drive channel/drive controller/device block size/raid controller).
The entire reason raid 6 was developed was due to this very issue.

So our databases are spread across multiple raid arrays.   If you remove
the ability to have multiple firebird database files, firebird will be
replaced by oracle or other platforms.

I like the concept of Firebird getting table spaces where we can specify
what those spaces are used for.
I would also like Firebird to keep the ability for those individual table
spaces to be held in multiple files.

regards

Dalton

On 3 March 2016 at 09:02, Vlad Khorsun  wrote:

> 03.03.2016 15:36, Dimitry Sibiryakov wrote:
> > 03.03.2016 13:38, Vlad Khorsun wrote:
> >>> May be transaction counters in each header written on each commit that
> change the
>  tablespace can work better?
> >>  This will create crazy overhead
> >
> > Yes. But aren't tablespaces considered for enterprise usage where IO
> for every
> > tablespace file is independent? In this case two writes have the same
> speed as one.
>
>If you have no better arguments, let's not create circus in every
> thread you are
> taking part.
>
> Vlad
>
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
>
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 15:07, Vlad Khorsun wrote:
>> > Are you sure that lock manager doesn't work with data of one or two 
>> > bytes long? At
>> >these sizes the function had a weak spot without these ifs.
> Perfect. All other 99% cases (with key >= 4 bytes) must pay for nothing.

   Tests shown no drawback for longer data.

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Vlad Khorsun]

03.03.2016 15:32, Dimitry Sibiryakov wrote:
> 03.03.2016 12:56, Vlad Khorsun wrote:
>>>  If you mean overall profit under multiuser load, then - no, I have no 
>>> numbers for that.
>>
>> Perfect
>
> If you have at hand server with dozen of cores and test suite you are 
> welcome to find
> it out yourself.

   It is you, who offer patch and should proof it is have value.

> I wouldn't expect much from lock manager, but hash joins can get 
> significant boost,
> especially on long keys.

   You don't need "server with dozen of cores" to check it, isn't is ?

>> PS There is no need in first two if's in CRC32C()
>
> Are you sure that lock manager doesn't work with data of one or two bytes 
> long? At
> these sizes the function had a weak spot without these ifs.

   Perfect. All other 99% cases (with key >= 4 bytes) must pay for nothing.

Vlad

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Wire protocol changes - was: Record level compression for V4

[Alex Peshkoff]

On 03/03/2016 03:10 PM, Mark Rotteveel wrote:
> On 2016-03-03 12:44, Alex Peshkoff wrote:
>> On 03/03/2016 12:26 PM, Mark Rotteveel wrote:
>>
>>> But my main concern
>>> has to do with my experience with wire protocol changes in Firebird:
>>> they are usually poorly documented,
>> Poorly is incorrect word here, remote protocol is not documented at
>> all
>> and (as far as I know) was never documented.
>> Except comments in sources.
> There is the wireprotocol documentation that was written by Carlos
> Guzman Alvarez when he did the Firebird .net implementation. I have
> since made some additions as part of my work with adding v11 and v12 to
> Jaybird. However, it is mostly the layout of the messages, and it misses
> a lot of context, expectations and meaning. So it could use some
> fleshing out with more information.
>
> There doesn't seem to be a recent build on the website, but an old
> build can be found in
> https://github.com/cincuranet/FirebirdSql.Data.FirebirdClient/blob/master/Documentation/FirebirdWireProtocol.pdf

Sorry, it's not protocol description - just rewritten in another form 
remote/protocol.h with parts of DPB tags list. _That_ quality document 
may be written in a day, but I'm sure you need something better...


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Improve compilation on OS X El Capitan

[marius adrian popa]

>From what i know current version of LibreOffice from git compiles just fine
on El Capitan (Firebird 2.5.5 with a few patches for c++11/14 compilers)

you can inspect the latest changes here
https://cgit.freedesktop.org/libreoffice/core/log/external/firebird

On Thu, Mar 3, 2016 at 7:30 AM, Quentin Pradet 
wrote:

> Thank you Paul!
>
> I know nothing about LibreOffice and FirebirdSQL, I was just trying to
> compile LibreOffice on my system, and I'm now trying to compile Firebird
> itself but this file does not appear to be included in LibreOffice so I'm
> probably wasting our time.
>
> Anyway, for LibreOffice, it looks like System Integrity Protection is
> getting in the way. I stumbled upon a hack from a LibreOffice contributor
> who went to great lengths to try to compile Firebird as part of LibreOffice
> by compiling bash just for Firebird:
> https://cgit.freedesktop.org/libreoffice/core/commit/?id=ce170cf1f264c69e91cde268ee490584c8fbcd04
>
> If and when I have more informations, I'll share to see if I can help to
> make things better.
>
> On Wed, Mar 2, 2016 at 7:24 PM, Paul Beach  wrote:
>
>> <>
>> I'm trying to compile Firebird on OS X El Capitan to understand why it
>> does not compile in LibreOffice [0]. But I can't compile it directly
>> either. I had issues with ICU configuration, but at least this issue
>> can be fixed directly in the code: CTest.c from cloop includes
>>  but this is deprecated and not needed in this file.
>>
>> [0]
>> https://lists.freedesktop.org/archives/libreoffice/2016-February/073399.html
>>
>> Here's the patch:
>>
>> Index: extern/cloop/src/tests/test1/CTest.c
>> ===
>> --- extern/cloop/src/tests/test1/CTest.c(revision 63007)
>> +++ extern/cloop/src/tests/test1/CTest.c(working copy)
>> @@ -20,7 +20,6 @@
>>   */
>>
>>  #include "CalcCApi.h"
>> -#include 
>>  #include 
>>
>>  #ifdef WIN32>>
>>
>> Committed into trunk and B3_0_Release.
>>
>> Paul
>>
>
>
>
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
> Firebird-Devel mailing list, web interface at
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
>
>
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Wire protocol changes - was: Record level compression for V4

[Mark Rotteveel]

On 2016-03-03 12:44, Alex Peshkoff wrote:
> On 03/03/2016 12:26 PM, Mark Rotteveel wrote:
>
>> But my main concern
>> has to do with my experience with wire protocol changes in Firebird:
>> they are usually poorly documented,
>
> Poorly is incorrect word here, remote protocol is not documented at 
> all
> and (as far as I know) was never documented.
> Except comments in sources.

There is the wireprotocol documentation that was written by Carlos 
Guzman Alvarez when he did the Firebird .net implementation. I have 
since made some additions as part of my work with adding v11 and v12 to 
Jaybird. However, it is mostly the layout of the messages, and it misses 
a lot of context, expectations and meaning. So it could use some 
fleshing out with more information.

There doesn't seem to be a recent build on the website, but an old 
build can be found in 
https://github.com/cincuranet/FirebirdSql.Data.FirebirdClient/blob/master/Documentation/FirebirdWireProtocol.pdf

It is in the manual project under 
/src/docs/firebirddocs/wireprotocol.xml

>> when you have to
>> reimplement in a different language (or find the right library to do 
>> the
>> work for you). That is why I'm stressing that things need to be 
>> specced
>> and documented when it involves the wire protocol.
>
> I plan to write such document - unfortunately can't do it soon. But
> please do not expect hard protocol description regarding initial
> handshake - with various authentication and wire encryption plugins 
> and
> related configuration settings (with encryption turned off less
> round-trips may be needed for handshake) such straightforward
> description of handshake protocol just can not exist.

I think that should be possible, otherwise the current implementation 
would be an 'accident' that it even works and there would be no 
guarantee it continues to work in the future when new changes are 
introduced (or new plugins added). It might be hard to write the 
expected state transitions and conditions and think of all corner cases 
after the fact, but I don't think it impossible to write the main 
flow(s).

In my opinion such things should be thought out first, reviewed and 
critiqued by others (eg us driver implementers), and then implemented, 
with maybe a number of iterations/revisions to fix things that turned 
out less straightforward when implementing.

Mark

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Browse information in VS 2010

[Dmitry Yemanov]

03.03.2016 13:28, Dimitry Sibiryakov wrote:
>
> Do anybody use subj?

Yes, me (sometimes).


Dmitry


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Vlad Khorsun]

03.03.2016 9:04, Dmitry Yemanov wrote:
> 03.03.2016 09:48, Dmitry Yemanov wrote:
>>
>>> Blobs could be moved into separate tablespace. It could make backup of
>>> "data" tablespace faster and smaller. We can even think about "offline"
>>> tablespace.
>>
>> Really good idea. Maybe more important for an average FB user than the
>> other usage cases.
>
> BTW, if blob is declared as belonging to some tablespace, should level-0
> blobs be also moved to its own tablespace or stored in the table tablespace?

   It depends on how we going to support tablespace for blobs. At first look
it could be an option in CREATE\ALTER TABLE statement and we could define
exact behavior there (if needed).

> And another question: does RDB$PAGES need to include the pagespace ID or
> we never allow tablespaces for TIP, generator pages and system tables?

   I think all system data should reside at system (default) tablespace.

> Pagespace ID for PP/IRP can be retrieved from RDB$ tables based on
> relation_id, so I don't see a problem to store just the "local" page number.

   Agree

Regards,
Vlad


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Browse information in VS 2010

[Vlad Khorsun]

03.03.2016 12:55, Dimitry Sibiryakov wrote:
> 03.03.2016 11:52, Vlad Khorsun wrote:
>>>   Do anybody use subj?
>>  Sure
>
> Don't you use VS 2013?..

   It have no impact on my opinion

Vlad


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] [Firebird-checkins] SF.net SVN: firebird:[63072] firebird/trunk/src

[Dimitry Sibiryakov]

03.03.2016 6:28, Dmitry Yemanov wrote:

I've asked for a particular change patch, you've provided 220KB of the
total tree diff to scan for the needed changes. If you want your patches
reviewed sooner, better start to respect other's time.


  Ok, here you have it, attached.

--
  WBR, SD.


Hash.7z
Description: Binary data
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Browse information in VS 2010

[Dimitry Sibiryakov]

03.03.2016 11:52, Vlad Khorsun wrote:
>>  Do anybody use subj?
> Sure

   Don't you use VS 2013?..

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Browse information in VS 2010

[Vlad Khorsun]

03.03.2016 12:28, Dimitry Sibiryakov wrote:
> Hello, All.
>
> Do anybody use subj?

   Sure

> Windows DDK 7.1 does not have browse symbol compiler for 64 bits. Because 
> of that build
> of 64 bits version fail with VS 2010 Express.

   IIRC, VS 2010 Express have no 64-bit compiler.

> Do you object if I commit changes in project files that turn generation 
> of browse
> symbols off?

It depends on who needs in this change. If VS 2010 Express really have no 
64-bit compiler
and you used some "method" to compile for x64, then i object, as this change is 
for you only.
If my memory is wrong and anybody could use VS 2010 Express to compile for x64 
- i have no
objection.

Vlad



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Firebird-devel] Browse information in VS 2010

[Dimitry Sibiryakov]

   Hello, All.

   Do anybody use subj?
   Windows DDK 7.1 does not have browse symbol compiler for 64 bits. Because of 
that build 
of 64 bits version fail with VS 2010 Express.
   Do you object if I commit changes in project files that turn generation of 
browse 
symbols off?

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Dimitry Sibiryakov]

03.03.2016 11:10, Atri Sharma wrote:
>
> > I feel that tablespaces allow managing multiple independent identities 
> inside a database
> > conveniently, and making stuff like backups pretty fast and granular.
>
> How are you going to solve problem with inconsistency between 
> tablespaces?
>
> Which specific inconsistency can you see,please?

   If tablespace rstored from granular backup is older that the rest, it can 
have records 
versions created by rollbacked transactions, but in new TIP this transaction 
can be 
already sweeped and its state changed to "committed". Such versions will be 
resurrected.
   If reusing of records formats is implemented, records' layout in outdated 
tablespace 
will not match format with the same number.
   The same problems if a tablespace is newer that the rest.

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Atri Sharma]

>
> > I feel that tablespaces allow managing multiple independent identities
> inside a database
> > conveniently, and making stuff like backups pretty fast and granular.
>
>How are you going to solve problem with inconsistency between
> tablespaces?
>
>
>

Which specific inconsistency can you see,please?

-- 
Regards,

Atri
*l'apprenant*
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Dimitry Sibiryakov]

03.03.2016 10:33, Atri Sharma wrote:
> Personally, I don't care much about tablespaces per se. But I foresee
> that table partitioning could be stacked over the tablespace
> implementation and I like this idea.
>
>
>
> I feel that tablespaces allow managing multiple independent identities inside 
> a database
> conveniently, and making stuff like backups pretty fast and granular.

   How are you going to solve problem with inconsistency between tablespaces?

-- 
   WBR, SD.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Jiří Činčura]

> they are usually poorly documented, which creates a large burden on 
> driver implementers having to follow all intricacies of the code (in my 
> case in a language that I'm not fluent in) to understand what changes 
> they need to implement. Especially with code involving encryption or 
> compression, C++ code is not the most readable when you have to 
> reimplement in a different language (or find the right library to do the 
> work for you). That is why I'm stressing that things need to be specced 
> and documented when it involves the wire protocol.

I second that. Firebird core developers never considered much how some
changes will impact drivers or how some features might be aligned to fit
other environments. 

-- 
Mgr. Jiří Činčura
Independent IT Specialist

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Atri Sharma]

>
>
> And another question: does RDB$PAGES need to include the pagespace ID or
> we never allow tablespaces for TIP, generator pages and system tables?
> Pagespace ID for PP/IRP can be retrieved from RDB$ tables based on
> relation_id, so I don't see a problem to store just the "local" page
> number.
>
>
I am green to Firebird, but I think that sounds like a good idea, albeit
requires another redirection to find the actual ID.
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[Atri Sharma]

On Thu, Mar 3, 2016 at 11:01 AM, Dmitry Yemanov  wrote:

> 03.03.2016 00:23, Dmitry Yemanov wrote:
> >
> > Someone may think about per-tablespace physical backups and other
> > possible usage cases.
>
> Personally, I don't care much about tablespaces per se. But I foresee
> that table partitioning could be stacked over the tablespace
> implementation and I like this idea.
>


I feel that tablespaces allow managing multiple independent identities
inside a database conveniently, and making stuff like backups pretty fast
and granular.

>
>
>
--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] Record level compression for V4

[Mark Rotteveel]

On 2016-03-03 1:35, Slavomir Skopalik wrote:
> Hi,
> this is about record encoding .
> My private test with RLE + LZ4 shows, that from combination RLE, LZ4,
> RLE + LZ4 the last is the best.

I'm perfectly ok with doing that for records.

> I believe, that better record encoding will help wire transfer in any 
> case.
>
>  From other side, record encoding can be used in all situation (in
> average speed can be around 10-30% of memory move)
> but zlib over local network can be limitation.

Decompression in memory would be necessary when comparing values, on 
sorts, etc. So I think in memory, you would probably work with the 
uncompressed values.

As to the wire protocol: Compression over compression is usually 
inefficient and something I think should be avoided. But my main concern 
has to do with my experience with wire protocol changes in Firebird: 
they are usually poorly documented, which creates a large burden on 
driver implementers having to follow all intricacies of the code (in my 
case in a language that I'm not fluent in) to understand what changes 
they need to implement. Especially with code involving encryption or 
compression, C++ code is not the most readable when you have to 
reimplement in a different language (or find the right library to do the 
work for you). That is why I'm stressing that things need to be specced 
and documented when it involves the wire protocol.

Mark

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] GitHub

[Mark Rotteveel]

On 2016-03-03 1:19, Adriano dos Santos Fernandes wrote:
> Em 02/03/2016 11:14, Michal Kubecek escreveu:
>> On Wed, Mar 02, 2016 at 10:45:35AM -0300, Adriano dos Santos 
>> Fernandes wrote:
>>> Em 02/03/2016 09:31, Michal Kubecek escreveu:
 On Wed, Mar 02, 2016 at 02:18:27PM +0300, Dmitry Yemanov wrote:
>
> Firebird Project will be moving its repositories to GitHub soon.

 Is the fact that the git repository hasn't been updated since 
 Sunday
 (and therefore has no B3_0_Release branch) related to this 
 migration
 or is it a different problem?

>>>
>>> The fact is more this: :D
>>>
>>> 
>>> http://stackoverflow.com/questions/35726537/track-a-branch-in-a-bare-git-svn
>>
>> That was my first hypothesis. :-)
>>
>
> Now it's syncing again.

Funny thing, the guy who answered your question on SO is a colleague of 
mine :)

Mark

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[liviuslivius]

I see that something strange happend with my email.
It is broken :(

something happened in place
"ecod version." -< i do not write this and also at the end i see duplicate 
lines, cuts and other strange things. Very strange for me...
I paste below orginal message


please do not touch shadows.
Shadows are used by many organisations. I saw it in almost any of them where i 
do some job for it.
And someone in near past worked on extending its usage e.g. CORE-4955

I agree with you that already existing feature "multi file database" is really 
"Database spaces".
New feature should extend this functionality for "kind of data".

Then in future we will have:
- database spaces
- table spaces

But i also disagree removing "database spaces" and implement only "table 
spaces".
Both are usefull.
Of course this features can not be used in the same database at the same time.
Only one of them.

But sumarise - i suppose that table spaces are on very lover priority and in my 
personal opinion should not be considered for FB V4 only V5.
I suppose that much usefull things should be implemented first - like clustered 
indexes, table partitioning, partial indexes, cube, rollup, grouping sets, 
R-TREE ...

When i saw that you consider removing multifile database - i have created 
CORE-5125.
"Database spaces" and feature to "prevent write to database page new data". 
I mean that you restore database with above switch (it explicitly include 
USE_ALL_SPACE switch) and all existing pages are marked as "not modify for new 
data" (below i call it MARKED). 
When something is updated new record version go into new page, and in page 
marked, only allowed operation is remove old record version. 
And now all new/modified data go into new place - eg. into fast accessible 
storage. Old data are on some slow storage. 
I do not write it previously into the tracker because of below reason (look 
below).

But summarize - i suppose that table spaces are on very lover priority and in 
my personal opinion should not be considered for FB V4 only V5.
I suppose that much useful things should be implemented first - like clustered 
indexes, table partitioning, partial indexes, cube, rollup, grouping sets, 
R-TREE ...

Regards,
Karol Bieniaszewski

W dniu 2016-03-03 09:15:24 użytkownik liviuslivius 
 napisał:
> Hi,
> 
> please do not touch shadows.
> Shadows are used by many organisations. I saw it in almost any of them where 
> i do some job for it.
> And someone in near past worked on extending its usage e.g. CORE-4955
> 
> I agree with you that already existing feature "multi file database" is 
> really "Database spaces".
> New feature should extend this functionality for "kind of data".
> 
> Then in future we will have:
> - database spaces
> - table spaces
> 
> But i also disagree removing "database spaces" and implement only "table 
> spaces".
> Both are usefull.
> Of course this features can not be used in the same database at the same time.
> Only one of them.
> 
> ecod version.
> And now all new data go into new place - eg. into fast accessible storage. 
> Old data are on some slow storage.
> 
> But sumarise - i suppose that table spaces are on very lover priority and in 
> my personal opinion should not be considered for FB V4 only V5.
> I suppose that much usefull things should be implemented first - like 
> clustered indexes, table partitioning, partial indexes, cube, rollup, 
> grouping sets, R-TREE ...
> 
> When i saw that you consider removing multifile database - i have created 
> CORE-5125.
> I do not write it previously into the tracker because of below reason (look 
> below).
> 
> But summarize - i suppose that table spaces are on very lover priority and in 
> my personal opinion should not be considered for FB V4 only V5.
> I suppose that much useful things should be implemented first - like 
> clustered indexes, table partitioning, partial indexes, cube, rollup, 
> grouping sets, R-TREE ...
> 
> 
> regards,
> Karol Bieniaszewski
> 
> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
> Firebird-Devel mailing list, web interface at 
> https://lists.sourceforge.net/lists/listinfo/firebird-devel
> 




--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140

[Firebird-devel] [FB-Tracker] Created: (CORE-5125) Add possibility to mark all pages at e.g. restore time as not writable for new incoming data

[Karol Bieniaszewski (JIRA)]

Add possibility to mark all pages at e.g. restore time as not writable for new 
incoming data


 Key: CORE-5125
 URL: http://tracker.firebirdsql.org/browse/CORE-5125
 Project: Firebird Core
  Issue Type: New Feature
  Components: Engine, GBAK
Reporter: Karol Bieniaszewski


We have now possibility to divide database into multiple files (i call this 
"Database speces").
This feature extend usability of this feature. 

"Database spaces" and feature to "prevent write to database page new data".
I mean that you restore database with above switch (it explicitly include 
USE_ALL_SPACE switch) and all existing pages are marked as "not modify for new 
data" (below i call it MARKED).
When something is updated new record version go into new page, and in page 
marked, only allowed operation is remove old record version.
And now all new/modified data go into new place - eg. into fast accessible 
storage. Old data are on some slow storage.

sumarrize:
- flag for database page "prevent incoming new data"
- restore with switch "prevent incoming new data to all existing pages"
- engine should ommit pages marked for storing new data 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: 
http://tracker.firebirdsql.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira



--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

Re: [Firebird-devel] RFC: Tablespaces

[liviuslivius]

Hi,

please do not touch shadows.
Shadows are used by many organisations. I saw it in almost any of them where i 
do some job for it.
And someone in near past worked on extending its usage e.g. CORE-4955

I agree with you that already existing feature "multi file database" is really 
"Database spaces".
New feature should extend this functionality for "kind of data".

Then in future we will have:
- database spaces
- table spaces

But i also disagree removing "database spaces" and implement only "table 
spaces".
Both are usefull.
Of course this features can not be used in the same database at the same time.
Only one of them.

ecod version.
And now all new data go into new place - eg. into fast accessible storage. Old 
data are on some slow storage.

But sumarise - i suppose that table spaces are on very lover priority and in my 
personal opinion should not be considered for FB V4 only V5.
I suppose that much usefull things should be implemented first - like clustered 
indexes, table partitioning, partial indexes, cube, rollup, grouping sets, 
R-TREE ...

When i saw that you consider removing multifile database - i have created 
CORE-5125.
I do not write it previously into the tracker because of below reason (look 
below).

But summarize - i suppose that table spaces are on very lover priority and in 
my personal opinion should not be considered for FB V4 only V5.
I suppose that much useful things should be implemented first - like clustered 
indexes, table partitioning, partial indexes, cube, rollup, grouping sets, 
R-TREE ...


regards,
Karol Bieniaszewski

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel

[Firebird-devel] gpre and boolean fields

[Paul Beach]

>Do you agree that support for boolean fields in gpre is a good thing?

If somebody provides the support, then I am unlikely to complain, strangely 
enough
I do know a few users who continue to use gpre.

Regards
Paul 

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140
Firebird-Devel mailing list, web interface at 
https://lists.sourceforge.net/lists/listinfo/firebird-devel