Re: Command substitution with null bytes generates warning
On Wed, Oct 05, 2016 at 09:20:58AM -0400, Chet Ramey wrote: > Try the attached patch, which reduces the number of warnings to 1 per call > to command substitution. I don't agree with this approach -- I think you should be able to turn off the warning completely, so I am not interested in testing the patch. In the script where I ran into this problem, I simply suppressed the warning using 2>&-, and that's what I will continue to do if I will knowingly read files that contain null bytes. Eric
Re: Command substitution with null bytes generates warning
On 9/16/16 1:51 AM, Eric Pruitt wrote:
> Bash Version: 4.4
> Patch Level: 0
> Release Status: release
>
> Description:
> I have a script that execute `if [[ "$(<"/proc/$1/cmdline")" = tmux*
> ]];`.
> All /proc/*/cmdline include null bytes, and as of Bash 4.4, this
> results in
> a warning being spewed on stderr which did not happen in Bash 4.3.
>
> Repeat-By:
> echo "$(<"/proc/$$/cmdline")"
>
> Fix:
> Is this even an intentional change? I looked at some of the other
> internal_warning invocations, and they were commented out using "#if 0
> ...
> #endif."
Try the attached patch, which reduces the number of warnings to 1 per call
to command substitution.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
*** ../bash-4.4/subst.c 2016-08-30 16:46:38.0 -0400
--- subst.c 2016-09-26 10:20:19.0 -0400
***
*** 5932,5935
--- 5933,5937
int istring_index, istring_size, c, tflag, skip_ctlesc, skip_ctlnul;
ssize_t bufn;
+ int nullbyte;
istring = (char *)NULL;
***
*** 5939,5942
--- 5941,5946
skip_ctlesc |= *s == CTLESC, skip_ctlnul |= *s == CTLNUL;
+ nullbyte = 0;
+
/* Read the output of the command through the pipe. This may need to be
changed to understand multibyte characters in the future. */
***
*** 5957,5961
{
#if 1
! internal_warning ("%s", _("command substitution: ignored null byte in input"));
#endif
continue;
--- 5961,5969
{
#if 1
! if (nullbyte == 0)
! {
! internal_warning ("%s", _("command substitution: ignored null byte in input"));
! nullbyte = 1;
! }
#endif
continue;
Re: bash history with mixed timestamps slow and broken
On 9/25/16 6:39 PM, Chet Ramey wrote:
> On 9/24/16 2:17 PM, Hubert Schmid wrote:
>> Bash Version: 4.4
>> Patch Level: 0
>> Release Status: release
>>
>> Description:
>> If the history file (`.bash_history`) starts with a timestamp
>> (`HIST_TIMESTAMP_START`), and contains lines that have been written
>> without timestamps, then reading the history file is (a) very slow
>> because (b) consecutive lines without timestamps are merged into a
>> single history entry with quadratic complexity.
>>
>> Apparently, this problem didn't exist in the previous version (4.3).
>
> One of the most frequently-requested features for the bash and readline
> history implementations is a way to preserve multi-line commands across
> shell sessions. It's been the subject of numerous previous discussions
> on bug-bash.
>
> There hasn't been a good way to do that, since there isn't good support
> for it in the traditional readline (flat) history file format.
>
> I decided to implement a heuristic: if the history file starts with a
> timestamp, and the application using the history library -- in this case,
> bash -- has indicated that it's interested in writing timestamps to the
> history file (which is off by default), the history file reading code
> assumes that timestamps appear consistently in the history file and it can
> use them as markers to delimit commands.
>
> The appending behavior isn't really quadratic: the code simply keeps
> reallocating the line and appending to it. You can imagine how long it
> takes to append a million commands to a single buffer. You've managed
> to identify the most degenerate case.
Try the attached patch and see whether or not it improves the load time.
You're still going to end up with one very long history entry.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
*** ../bash-4.4/lib/readline/history.c 2015-12-28 13:50:31.0 -0500
--- lib/readline/history.c 2016-09-30 14:28:40.0 -0400
***
*** 417,426
{
HIST_ENTRY *hent;
! size_t newlen, curlen;
char *newline;
hent = the_history[which];
curlen = strlen (hent->line);
! newlen = curlen + strlen (line) + 2;
newline = realloc (hent->line, newlen);
if (newline)
--- 421,441
{
HIST_ENTRY *hent;
! size_t newlen, curlen, minlen;
char *newline;
hent = the_history[which];
curlen = strlen (hent->line);
! minlen = curlen + strlen (line) + 2; /* min space needed */
! if (curlen > 256) /* XXX - for now */
! {
! newlen = 512; /* now realloc in powers of 2 */
! /* we recalcluate every time; the operations are cheap */
! while (newlen < minlen)
! newlen <<= 1;
! }
! else
! newlen = minlen;
! /* Assume that realloc returns the same pointer and doesn't try a new
! alloc/copy if the new size is the same as the one last passed. */
newline = realloc (hent->line, newlen);
if (newline)
Re: "HISTSIZE=999999999" cause bash failure after lastest upgrade
On 9/25/16 6:59 PM, Chet Ramey wrote:
> On 9/25/16 3:51 PM, Sean Zha wrote:
>
>> Bash Version: 4.4
>> Patch Level: 0
>> Release Status: release
>>
>> Description:
>> I use a huge value for HISTSIZE (=9) to enable infinite
>> history items. The actural size of ~/.bash_history is only 4MB now.
>> Everything worked fine before the lastest upgrade. Now bash refuse
>> me to login due to memory allocation failure. After choosing
>> a smaller HISTSIZE, bash still eatup too much unnesssary memory.
>>
>> Repeat-By:
>> open a workable terminal with small HISTSIZE setting,
>> $ echo HISTSIZE=9 > ~/test.rc
>> $ bash --rcfile ~/test.rc
>>bash: xmalloc: cannot allocate 88 bytes (114688 bytes
>> allocated)
>
> Since you've specified the desired history size, bash tries to allocate
> enough entries to hold all of the entries. The assumption is that this
> will reduce the number of allocations/reallocations and the number of
> times the history list has to be copied as it's reallocated. I suppose
> I should cap the maximum value for which that occurs instead of trusting
> the supplied max number of entries.
Try the attached patch and see whether or not it improves things.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
*** ../bash-4.4/lib/readline/history.c 2015-12-28 13:50:31.0 -0500
--- lib/readline/history.c 2016-09-30 14:28:40.0 -0400
***
*** 58,61
--- 58,63
#define DEFAULT_HISTORY_INITIAL_SIZE 502
+ #define MAX_HISTORY_INITIAL_SIZE 8192
+
/* The number of slots to increase the_history by. */
#define DEFAULT_HISTORY_GROW_SIZE 50
***
*** 308,312
{
if (history_stifled && history_max_entries > 0)
! history_size = history_max_entries + 2;
else
history_size = DEFAULT_HISTORY_INITIAL_SIZE;
--- 310,316
{
if (history_stifled && history_max_entries > 0)
! history_size = (history_max_entries > MAX_HISTORY_INITIAL_SIZE)
! ? MAX_HISTORY_INITIAL_SIZE
! : history_max_entries + 2;
else
history_size = DEFAULT_HISTORY_INITIAL_SIZE;
