Re: Bootstrapping multiple cassandra nodes simultaneously in existing dc

[Bhuvan Rawal]

Thanks Jens! this was helpful, also to avoid pending compaction buildup
compaction throughput can be throttled higher. In our case however the
property batchlog throttle (batchlog_replay_throttle_in_kb) was being the
bottleneck increasing it to 10240k from default of 1024k reduced node
addition time by atleast a factor 3x. (from this it could be inferred that
batchlog is used during node addition).

An interesting hack for people on public cloud could be to get themselves a
higher cpu capacity machine during bootstrap and then downgrading it once
data is in place - as cpu essentially becomes bottleneck during node
addition.

I could find a property in docs -  consistent.rangemovement, setting it to
false which allows multiple node addition simultaneously.


*JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false"*

I could test with this property in a test cluster and things appeared fine.
This constraint seems to be introduced in CASSANDRA-2434
. If we talk of
today, what could be the possible implication of multiple node addition
simultaneously, suppose if 2 min rule is taken into accord and a repair
that works!

Regards,
Bhuvan

On Mon, Sep 12, 2016 at 2:56 AM, Jens Rantil  wrote:

> Yes. `nodetool setstreamthroughput` is your friend.
>
>
> On Sunday, September 11, 2016, sai krishnam raju potturi <
> pskraj...@gmail.com> wrote:
>
>> Make sure there is no spike in the load-avg on the existing nodes, as
>> that might affect your application read request latencies.
>>
>> On Sun, Sep 11, 2016, 17:10 Jens Rantil  wrote:
>>
>>> Hi Bhuvan,
>>>
>>> I have done such expansion multiple times and can really recommend
>>> bootstrapping a new DC and pointing your clients to it. The process is so
>>> much faster and the documentation you referred to has worked out fine for
>>> me.
>>>
>>> Cheers,
>>> Jens
>>>
>>>
>>> On Sunday, September 11, 2016, Bhuvan Rawal  wrote:
>>>
 Hi,

 We are running Cassandra 3.6 and want to bump up Cassandra nodes in an
 existing datacenter from 3 to 12 (plan to move to r3.xlarge machines to
 leverage more memory instead of m4.2xlarge). Bootstrapping a node would
 take 7-8 hours.

 If this activity is performed serially then it will take 5-6 days. I
 had a look at CASSANDRA-7069
  and a bit of
 discussion in the past at - http://grokbase.com/t/cassan
 dra/user/147gcqvybg/adding-more-nodes-into-the-cluster. Wanted to know
 if the limitation is still applicable and race condition could occur in 3.6
 version.

 If this is not the case can we add a new datacenter as mentioned here
 opsAddDCToCluster
 
  and
 bootstrap multiple nodes simultaneously by keeping auto_bootstrap false in
 cassandra.yaml and rebuilding nodes simultaneously in the new dc?


 Thanks & Regards,
 Bhuvan

>>>
>>>
>>> --
>>> Jens Rantil
>>> Backend engineer
>>> Tink AB
>>>
>>> Email: jens.ran...@tink.se
>>> Phone: +46 708 84 18 32
>>> Web: www.tink.se
>>>
>>> Facebook  Linkedin
>>> 
>>>  Twitter 
>>>
>>>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook  Linkedin
> 
>  Twitter 
>
>

Re: Why does `now()` produce different times within the same query?

[Jeff Jirsa]

On 2016-12-03 08:44 (-0800), Edward Capriolo  wrote: 
> On Sat, Dec 3, 2016 at 11:01 AM, Edward Capriolo 
> wrote:
> 
> >
> >
> >  A new unique timeuuid (at the time where the statement using it is
> > executed).
> >
> > Indicates that each statement has one unique time uuid. Calling the udf
> > twice in one statement and getting different results dissagrees with the
> > documentation.
> >
> 
> https://issues.apache.org/jira/browse/CASSANDRA-12989
> 

Reasonable change to me. doc change committed to trunk - it'll make its way to 
the site soon'ish.

Re: Why does `now()` produce different times within the same query?

[Edward Capriolo]

On Sat, Dec 3, 2016 at 11:01 AM, Edward Capriolo 
wrote:

>
>
> On Saturday, December 3, 2016, Edward Capriolo 
> wrote:
>
>>
>>
>> On Saturday, December 3, 2016, Jonathan Haddad  wrote:
>>
>>> That isn't what the original thread is about. The thread is about the
>>> timestamp portion of the UUID being different.
>>>
>>> Having UUID() return the same thing for all rows in a batch would be the
>>> unexpected thing virtually every time.
>>> On Sat, Dec 3, 2016 at 7:09 AM Edward Capriolo 
>>> wrote:
>>>


 On Friday, December 2, 2016, Jonathan Haddad  wrote:

> This isn't about using the same UUID though. It's about the timestamp
> bits in the UUID.
>
> What the use case is for generating multiple UUIDs in a single row?
> Why do you need to extract the timestamp out of both?
> On Fri, Dec 2, 2016 at 10:24 AM Edward Capriolo 
> wrote:
>
>>
>> On Thu, Dec 1, 2016 at 11:09 AM, Sylvain Lebresne <
>> sylv...@datastax.com> wrote:
>>
>>> On Thu, Dec 1, 2016 at 4:44 PM, Edward Capriolo <
>>> edlinuxg...@gmail.com> wrote:
>>>

 I am not sure you saw my reply on thread but I believe everyone's
 needs can be met I will copy that here:

>>>
>>> I saw it, but the real problem that was raised initially was not
>>> that of UDF and of allowing both behavior. It's a matter of people being
>>> confused by the behavior of a non-UDF function, now(), and suggesting it
>>> should be changed.
>>>
>>> The Hive idea is interesting I guess, and we can switch to
>>> discussing that, but it's a different problem really and I'm not a fond 
>>> of
>>> derailing threads. I will just note though that if we're not talking 
>>> about
>>> a confusion issue but rather how to get a timeuuid to be fixed within a
>>> statement, then there is much much more trivial solution: generate it
>>> client side. The `now()` function is a small convenience but there is
>>> nothing you cannot do without it client side, and that actually 
>>> basically
>>> stands for almost any use of (non aggregate) function in Cassandra
>>> currently.
>>>
>>>


 "Food for thought: Hive's UDFs introduced an annotation
 @UDFType(deterministic = false)

 http://dmtolpeko.com/2014/10/15/invoking-stateful-udf-at-map
 -and-reduce-side-in-hive/

 The effect is the query planner can see when such a UDF is in use
 and determine the value once at the start of a very long query."

 Essentially hive had a similar if not identical problem, during a
 long running distributed process like map/reduce some users wanted the
 semantics of:

 1) Each call should have a new timestamps

 While other users wanted the semantics of:

 2) Each call should generate the same timestamp

 The solution implemented was to add an annotation to udf such that
 the query planner would pick up the annotation and act accordingly.

 (Here is a related issue https://issues.apache.or
 g/jira/browse/HIVE-1986

 As a result you can essentially implement two UDFS

 @UDFType(deterministic = false)
 public class UDFNow

 and for the other people

 @UDFType(deterministic = true)
 public class UDFNowOnce extends UDFNow

 Both user cases are met in a sensible way.

>>>
>>>
>> The `now()` function is a small convenience but there is nothing you
>> cannot do without it client side, and that actually basically stands for
>> almost any use of (non aggregate) function in Cassandra currently.
>>
>> Casandra's changing philosophy over which entity should create such
>> information client/server/driver does not make this problem easy.
>>
>> If you take into account that you have users who do not understand
>> all the intricacy of uuid the problem is compounded. IE How does one
>> generate a UUID each c#, python, java etc? with the 47 random bits of bla
>> bla. That is not super easy information to find. Maybe you find a stack
>> overflow post that actually gives bad advice etc.
>>
>> Many times in Cassandra you are using a uuid because you do not have
>> a unique key in the insert and you wish to create one. If you are 
>> inserting
>> more then a single record using that same UUID and you do not want the
>> burden of wanting to do it yourself you would have to do 
>> write>>read>>write
>> which is an anti-pattern.
>>
>
 Not multiple ids for a single row. The same id for multiple inserts in
 a batch.

Re: Why does `now()` produce different times within the same query?

[Edward Capriolo]

On Saturday, December 3, 2016, Edward Capriolo 
wrote:

>
>
> On Saturday, December 3, 2016, Jonathan Haddad  > wrote:
>
>> That isn't what the original thread is about. The thread is about the
>> timestamp portion of the UUID being different.
>>
>> Having UUID() return the same thing for all rows in a batch would be the
>> unexpected thing virtually every time.
>> On Sat, Dec 3, 2016 at 7:09 AM Edward Capriolo 
>> wrote:
>>
>>>
>>>
>>> On Friday, December 2, 2016, Jonathan Haddad  wrote:
>>>
 This isn't about using the same UUID though. It's about the timestamp
 bits in the UUID.

 What the use case is for generating multiple UUIDs in a single row? Why
 do you need to extract the timestamp out of both?
 On Fri, Dec 2, 2016 at 10:24 AM Edward Capriolo 
 wrote:

>
> On Thu, Dec 1, 2016 at 11:09 AM, Sylvain Lebresne <
> sylv...@datastax.com> wrote:
>
>> On Thu, Dec 1, 2016 at 4:44 PM, Edward Capriolo <
>> edlinuxg...@gmail.com> wrote:
>>
>>>
>>> I am not sure you saw my reply on thread but I believe everyone's
>>> needs can be met I will copy that here:
>>>
>>
>> I saw it, but the real problem that was raised initially was not that
>> of UDF and of allowing both behavior. It's a matter of people being
>> confused by the behavior of a non-UDF function, now(), and suggesting it
>> should be changed.
>>
>> The Hive idea is interesting I guess, and we can switch to discussing
>> that, but it's a different problem really and I'm not a fond of derailing
>> threads. I will just note though that if we're not talking about a
>> confusion issue but rather how to get a timeuuid to be fixed within a
>> statement, then there is much much more trivial solution: generate it
>> client side. The `now()` function is a small convenience but there is
>> nothing you cannot do without it client side, and that actually basically
>> stands for almost any use of (non aggregate) function in Cassandra
>> currently.
>>
>>
>>>
>>>
>>> "Food for thought: Hive's UDFs introduced an annotation
>>> @UDFType(deterministic = false)
>>>
>>> http://dmtolpeko.com/2014/10/15/invoking-stateful-udf-at-map
>>> -and-reduce-side-in-hive/
>>>
>>> The effect is the query planner can see when such a UDF is in use
>>> and determine the value once at the start of a very long query."
>>>
>>> Essentially hive had a similar if not identical problem, during a
>>> long running distributed process like map/reduce some users wanted the
>>> semantics of:
>>>
>>> 1) Each call should have a new timestamps
>>>
>>> While other users wanted the semantics of:
>>>
>>> 2) Each call should generate the same timestamp
>>>
>>> The solution implemented was to add an annotation to udf such that
>>> the query planner would pick up the annotation and act accordingly.
>>>
>>> (Here is a related issue https://issues.apache.or
>>> g/jira/browse/HIVE-1986
>>>
>>> As a result you can essentially implement two UDFS
>>>
>>> @UDFType(deterministic = false)
>>> public class UDFNow
>>>
>>> and for the other people
>>>
>>> @UDFType(deterministic = true)
>>> public class UDFNowOnce extends UDFNow
>>>
>>> Both user cases are met in a sensible way.
>>>
>>
>>
> The `now()` function is a small convenience but there is nothing you
> cannot do without it client side, and that actually basically stands for
> almost any use of (non aggregate) function in Cassandra currently.
>
> Casandra's changing philosophy over which entity should create such
> information client/server/driver does not make this problem easy.
>
> If you take into account that you have users who do not understand all
> the intricacy of uuid the problem is compounded. IE How does one generate 
> a
> UUID each c#, python, java etc? with the 47 random bits of bla bla. That 
> is
> not super easy information to find. Maybe you find a stack overflow post
> that actually gives bad advice etc.
>
> Many times in Cassandra you are using a uuid because you do not have a
> unique key in the insert and you wish to create one. If you are inserting
> more then a single record using that same UUID and you do not want the
> burden of wanting to do it yourself you would have to do 
> write>>read>>write
> which is an anti-pattern.
>

>>> Not multiple ids for a single row. The same id for multiple inserts in a
>>> batch.
>>>
>>> For example lets say I have an application where my data has no unique
>>> key.
>>>
>>> Table poke
>>> Poker, pokee, time
>>>
>>> Suppose i consume pokes from kafka build 

Re: Why does `now()` produce different times within the same query?

[Edward Capriolo]

On Saturday, December 3, 2016, Jonathan Haddad  wrote:

> That isn't what the original thread is about. The thread is about the
> timestamp portion of the UUID being different.
>
> Having UUID() return the same thing for all rows in a batch would be the
> unexpected thing virtually every time.
> On Sat, Dec 3, 2016 at 7:09 AM Edward Capriolo  > wrote:
>
>>
>>
>> On Friday, December 2, 2016, Jonathan Haddad > > wrote:
>>
>>> This isn't about using the same UUID though. It's about the timestamp
>>> bits in the UUID.
>>>
>>> What the use case is for generating multiple UUIDs in a single row? Why
>>> do you need to extract the timestamp out of both?
>>> On Fri, Dec 2, 2016 at 10:24 AM Edward Capriolo 
>>> wrote:
>>>

 On Thu, Dec 1, 2016 at 11:09 AM, Sylvain Lebresne  wrote:

> On Thu, Dec 1, 2016 at 4:44 PM, Edward Capriolo  > wrote:
>
>>
>> I am not sure you saw my reply on thread but I believe everyone's
>> needs can be met I will copy that here:
>>
>
> I saw it, but the real problem that was raised initially was not that
> of UDF and of allowing both behavior. It's a matter of people being
> confused by the behavior of a non-UDF function, now(), and suggesting it
> should be changed.
>
> The Hive idea is interesting I guess, and we can switch to discussing
> that, but it's a different problem really and I'm not a fond of derailing
> threads. I will just note though that if we're not talking about a
> confusion issue but rather how to get a timeuuid to be fixed within a
> statement, then there is much much more trivial solution: generate it
> client side. The `now()` function is a small convenience but there is
> nothing you cannot do without it client side, and that actually basically
> stands for almost any use of (non aggregate) function in Cassandra
> currently.
>
>
>>
>>
>> "Food for thought: Hive's UDFs introduced an annotation  
>> @UDFType(deterministic
>> = false)
>>
>> http://dmtolpeko.com/2014/10/15/invoking-stateful-udf-at-
>> map-and-reduce-side-in-hive/
>>
>> The effect is the query planner can see when such a UDF is in use and
>> determine the value once at the start of a very long query."
>>
>> Essentially hive had a similar if not identical problem, during a
>> long running distributed process like map/reduce some users wanted the
>> semantics of:
>>
>> 1) Each call should have a new timestamps
>>
>> While other users wanted the semantics of:
>>
>> 2) Each call should generate the same timestamp
>>
>> The solution implemented was to add an annotation to udf such that
>> the query planner would pick up the annotation and act accordingly.
>>
>> (Here is a related issue https://issues.apache.
>> org/jira/browse/HIVE-1986
>>
>> As a result you can essentially implement two UDFS
>>
>> @UDFType(deterministic = false)
>> public class UDFNow
>>
>> and for the other people
>>
>> @UDFType(deterministic = true)
>> public class UDFNowOnce extends UDFNow
>>
>> Both user cases are met in a sensible way.
>>
>
>
 The `now()` function is a small convenience but there is nothing you
 cannot do without it client side, and that actually basically stands for
 almost any use of (non aggregate) function in Cassandra currently.

 Casandra's changing philosophy over which entity should create such
 information client/server/driver does not make this problem easy.

 If you take into account that you have users who do not understand all
 the intricacy of uuid the problem is compounded. IE How does one generate a
 UUID each c#, python, java etc? with the 47 random bits of bla bla. That is
 not super easy information to find. Maybe you find a stack overflow post
 that actually gives bad advice etc.

 Many times in Cassandra you are using a uuid because you do not have a
 unique key in the insert and you wish to create one. If you are inserting
 more then a single record using that same UUID and you do not want the
 burden of wanting to do it yourself you would have to do write>>read>>write
 which is an anti-pattern.

>>>
>> Not multiple ids for a single row. The same id for multiple inserts in a
>> batch.
>>
>> For example lets say I have an application where my data has no unique
>> key.
>>
>> Table poke
>> Poker, pokee, time
>>
>> Suppose i consume pokes from kafka build a batch of 30k and insert them.
>> You probably want to denormalize into two tables:
>> Primary key (poker, time)
>> Primary key (pokee,time)
>>
>> It makes 

Re: Why does `now()` produce different times within the same query?

[Jonathan Haddad]

That isn't what the original thread is about. The thread is about the
timestamp portion of the UUID being different.

Having UUID() return the same thing for all rows in a batch would be the
unexpected thing virtually every time.
On Sat, Dec 3, 2016 at 7:09 AM Edward Capriolo 
wrote:

>
>
> On Friday, December 2, 2016, Jonathan Haddad  wrote:
>
> This isn't about using the same UUID though. It's about the timestamp bits
> in the UUID.
>
> What the use case is for generating multiple UUIDs in a single row? Why do
> you need to extract the timestamp out of both?
> On Fri, Dec 2, 2016 at 10:24 AM Edward Capriolo 
> wrote:
>
>
> On Thu, Dec 1, 2016 at 11:09 AM, Sylvain Lebresne 
> wrote:
>
> On Thu, Dec 1, 2016 at 4:44 PM, Edward Capriolo 
> wrote:
>
>
> I am not sure you saw my reply on thread but I believe everyone's needs
> can be met I will copy that here:
>
>
> I saw it, but the real problem that was raised initially was not that of
> UDF and of allowing both behavior. It's a matter of people being confused
> by the behavior of a non-UDF function, now(), and suggesting it should be
> changed.
>
> The Hive idea is interesting I guess, and we can switch to discussing
> that, but it's a different problem really and I'm not a fond of derailing
> threads. I will just note though that if we're not talking about a
> confusion issue but rather how to get a timeuuid to be fixed within a
> statement, then there is much much more trivial solution: generate it
> client side. The `now()` function is a small convenience but there is
> nothing you cannot do without it client side, and that actually basically
> stands for almost any use of (non aggregate) function in Cassandra
> currently.
>
>
>
>
> "Food for thought: Hive's UDFs introduced an annotation  
> @UDFType(deterministic
> = false)
>
>
> http://dmtolpeko.com/2014/10/15/invoking-stateful-udf-at-map-and-reduce-side-in-hive/
>
> The effect is the query planner can see when such a UDF is in use and
> determine the value once at the start of a very long query."
>
> Essentially hive had a similar if not identical problem, during a long
> running distributed process like map/reduce some users wanted the semantics
> of:
>
> 1) Each call should have a new timestamps
>
> While other users wanted the semantics of:
>
> 2) Each call should generate the same timestamp
>
> The solution implemented was to add an annotation to udf such that the
> query planner would pick up the annotation and act accordingly.
>
> (Here is a related issue https://issues.apache.org/jira/browse/HIVE-1986
>
> As a result you can essentially implement two UDFS
>
> @UDFType(deterministic = false)
> public class UDFNow
>
> and for the other people
>
> @UDFType(deterministic = true)
> public class UDFNowOnce extends UDFNow
>
> Both user cases are met in a sensible way.
>
>
>
> The `now()` function is a small convenience but there is nothing you
> cannot do without it client side, and that actually basically stands for
> almost any use of (non aggregate) function in Cassandra currently.
>
> Casandra's changing philosophy over which entity should create such
> information client/server/driver does not make this problem easy.
>
> If you take into account that you have users who do not understand all the
> intricacy of uuid the problem is compounded. IE How does one generate a
> UUID each c#, python, java etc? with the 47 random bits of bla bla. That is
> not super easy information to find. Maybe you find a stack overflow post
> that actually gives bad advice etc.
>
> Many times in Cassandra you are using a uuid because you do not have a
> unique key in the insert and you wish to create one. If you are inserting
> more then a single record using that same UUID and you do not want the
> burden of wanting to do it yourself you would have to do write>>read>>write
> which is an anti-pattern.
>
>
> Not multiple ids for a single row. The same id for multiple inserts in a
> batch.
>
> For example lets say I have an application where my data has no unique
> key.
>
> Table poke
> Poker, pokee, time
>
> Suppose i consume pokes from kafka build a batch of 30k and insert them.
> You probably want to denormalize into two tables:
> Primary key (poker, time)
> Primary key (pokee,time)
>
> It makes sense that they all have the same uuid if you want it to be the
> uuid of the batch. This would make it easy to correlate all the events.
> Easy to delete them all as well.
>
> The do it client side argument is totally valid, but has been a
> justification for not adding features many of which are eventually added
> anyway.
>
>
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>

Re: Why does `now()` produce different times within the same query?

[Edward Capriolo]

On Friday, December 2, 2016, Jonathan Haddad  wrote:

> This isn't about using the same UUID though. It's about the timestamp bits
> in the UUID.
>
> What the use case is for generating multiple UUIDs in a single row? Why do
> you need to extract the timestamp out of both?
> On Fri, Dec 2, 2016 at 10:24 AM Edward Capriolo  > wrote:
>
>>
>> On Thu, Dec 1, 2016 at 11:09 AM, Sylvain Lebresne > > wrote:
>>
>>> On Thu, Dec 1, 2016 at 4:44 PM, Edward Capriolo >> > wrote:
>>>

 I am not sure you saw my reply on thread but I believe everyone's needs
 can be met I will copy that here:

>>>
>>> I saw it, but the real problem that was raised initially was not that of
>>> UDF and of allowing both behavior. It's a matter of people being confused
>>> by the behavior of a non-UDF function, now(), and suggesting it should be
>>> changed.
>>>
>>> The Hive idea is interesting I guess, and we can switch to discussing
>>> that, but it's a different problem really and I'm not a fond of derailing
>>> threads. I will just note though that if we're not talking about a
>>> confusion issue but rather how to get a timeuuid to be fixed within a
>>> statement, then there is much much more trivial solution: generate it
>>> client side. The `now()` function is a small convenience but there is
>>> nothing you cannot do without it client side, and that actually basically
>>> stands for almost any use of (non aggregate) function in Cassandra
>>> currently.
>>>
>>>


 "Food for thought: Hive's UDFs introduced an annotation  
 @UDFType(deterministic
 = false)

 http://dmtolpeko.com/2014/10/15/invoking-stateful-udf-at-
 map-and-reduce-side-in-hive/

 The effect is the query planner can see when such a UDF is in use and
 determine the value once at the start of a very long query."

 Essentially hive had a similar if not identical problem, during a long
 running distributed process like map/reduce some users wanted the semantics
 of:

 1) Each call should have a new timestamps

 While other users wanted the semantics of:

 2) Each call should generate the same timestamp

 The solution implemented was to add an annotation to udf such that the
 query planner would pick up the annotation and act accordingly.

 (Here is a related issue https://issues.apache.
 org/jira/browse/HIVE-1986

 As a result you can essentially implement two UDFS

 @UDFType(deterministic = false)
 public class UDFNow

 and for the other people

 @UDFType(deterministic = true)
 public class UDFNowOnce extends UDFNow

 Both user cases are met in a sensible way.

>>>
>>>
>> The `now()` function is a small convenience but there is nothing you
>> cannot do without it client side, and that actually basically stands for
>> almost any use of (non aggregate) function in Cassandra currently.
>>
>> Casandra's changing philosophy over which entity should create such
>> information client/server/driver does not make this problem easy.
>>
>> If you take into account that you have users who do not understand all
>> the intricacy of uuid the problem is compounded. IE How does one generate a
>> UUID each c#, python, java etc? with the 47 random bits of bla bla. That is
>> not super easy information to find. Maybe you find a stack overflow post
>> that actually gives bad advice etc.
>>
>> Many times in Cassandra you are using a uuid because you do not have a
>> unique key in the insert and you wish to create one. If you are inserting
>> more then a single record using that same UUID and you do not want the
>> burden of wanting to do it yourself you would have to do write>>read>>write
>> which is an anti-pattern.
>>
>
Not multiple ids for a single row. The same id for multiple inserts in a
batch.

For example lets say I have an application where my data has no unique key.

Table poke
Poker, pokee, time

Suppose i consume pokes from kafka build a batch of 30k and insert them.
You probably want to denormalize into two tables:
Primary key (poker, time)
Primary key (pokee,time)

It makes sense that they all have the same uuid if you want it to be the
uuid of the batch. This would make it easy to correlate all the events.
Easy to delete them all as well.

The do it client side argument is totally valid, but has been a
justification for not adding features many of which are eventually added
anyway.




-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.