Re: performance bug of [[ $x ]]

[Peng Yu]

My OS is Mac OS X. I don't have perf. Is it only on linux? Could you
show me the output of your perf?

On 3/7/20, Chris Down  wrote:
> Peng Yu writes:
>>Could you show me how you do the profiling for this specific case?
>>Based on what proof that you can conclude that it is not the `[[`
>>performance problem?
>
> Like I said, `perf` is perfectly adequate.
>
>  bash -c 'x=$(printf "%.sx" {1..1000}); perf record -g -p $$ & sleep
> 2; time [[ $x ]]'
>
> You might as well just use `:`.
>


-- 
Regards,
Peng

Re: performance bug of [[ $x ]]

[Peng Yu]

Could you show me how you do the profiling for this specific case?
Based on what proof that you can conclude that it is not the `[[`
performance problem?

On 3/7/20, Chris Down  wrote:
> Peng Yu writes:
>>[[ $x ]] just tests whether the variable $x is of length 0 or not. So
>>its performance should not depend on how long the variable is.
>
> Who said it has anything to do with the [[ builtin's performance? A shell
> does
> a lot more than just running one command.
>
> For this and the last issue you posted, you could quite trivially find out
> the
> answer yourself using `perf` or another profiler.
>


-- 
Regards,
Peng

performance bug of [[ $x ]]

[Peng Yu]

[[ $x ]] just tests whether the variable $x is of length 0 or not. So
its performance should not depend on how long the variable is.

But the following test case shows that the run time does depend on the
length of the variable.

Should it be considered as a performance bug of bash?

$ x=$(printf '%.sx' {1..100})
$ time [[ $x ]]

real0m0.004s
user0m0.003s
sys 0m0.000s
$ x=$(printf '%.sx' {1..1000})
$ time [[ $x ]]

real0m0.043s
user0m0.039s
sys 0m0.004s

-- 
Regards,
Peng

Performance bug of {1..1000000}?

[Peng Yu]

See the following run time comparison. {1..100} is slower than
$(seq 100).

Since seq involves an external program, I'd expect the latter to be
slower. But the comparison shows the opposite.

I guess seq did some optimization?

Can the performance of {1..100} be improved so that it is faster
than $(seq 100)?

$ time builtin printf %.sx {1..100}  > /dev/null

real0m2.614s
user0m2.361s
sys 0m0.166s
$ time builtin printf %.sx $(seq 100)  > /dev/null

real0m1.516s
user0m1.317s
sys 0m0.158s

-- 
Regards,
Peng

Use high bits of the raw random number?

[Peng Yu]

Hi,

Bash uses the low 16 bits for $RANDOM.

https://git.savannah.gnu.org/cgit/bash.git/tree/variables.c#n1321
https://git.savannah.gnu.org/cgit/bash.git/tree/variables.c#n1356

It seems that the high bits should be more random. If so, maybe the
high 16 bits should be kept if $RANDOM must stay in 16bits?

https://stackoverflow.com/questions/13104478/uniformity-of-random-numbers-taken-modulo-n

-- 
Regards,
Peng

About ARITH_CMD

[Peng Yu]

Hi,

yylex() still gives the token ARITH_CMD for the following command. The
error seems to be raised at the parsing stage. Shouldn't the error be
caught in the lexical analysis stage?

$ ((x = 10 + 5; ++x; echo $x))
bash: ((: x = 10 + 5; ++x: syntax error: invalid arithmetic operator
(error token is "; ++x")

Why the parsing of the arithmetic expression is in the lexical
analysis. Why not introduce token `((` and `))` and handle arithmetic
expression in the bison parsing code?

Also, I don't find that POSIX specifies `((`. (Let me know if I miss
anything.) If `((` is a bash-specific thing, why not allow it to
handle multiple arithmetic expressions instead of just one? Thanks.

http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html

-- 
Regards,
Peng

Why { is parsed differently depending on the context?

[Peng Yu]

Hi,

`echo {` treats `{` as WORD.

`{  echo; }` treats `{` as a token of `{`.

`{a` treats `{a` as a WORD.

I don't see the point why yylex() treat `{` context dependently.
Wouldn't it better just treat a bare `{` as a token of `{`?

What is the reasoning behind the current design of the syntax?

-- 
Regards,
Peng

What is the purpose of ASSIGNMENT_WORD?

[Peng Yu]

Hi,

I know that ASSIGNMENT_WORD in parse.y is for assignments like x=10.

But in the grammar rules. I don't see any difference between them in
terms of actions to take. Where is the code that deals with them
differently?

Also, why parse x=10 as a single token. Why not parse it as three
tokens "x" "=" "10"? Is it because one wants to control the complexity
of the grammar? Thanks.

https://github.com/mfragkoulis/bash/blob/master/parse.y#L710
simple_command_element: WORD
  { $$.word = $1; $$.redirect = 0; }
  | ASSIGNMENT_WORD
  { $$.word = $1; $$.redirect = 0; }

https://github.com/mfragkoulis/bash/blob/master/parse.y#L3098
  if (token_to_read == WORD || token_to_read == ASSIGNMENT_WORD)
  {
yylval.word = word_desc_to_read;
word_desc_to_read = (WORD_DESC *)NULL;
  }

https://github.com/mfragkoulis/bash/blob/master/parse.y#L5748
case WORD:
case ASSIGNMENT_WORD:
  if (yylval.word)
  t = savestring (yylval.word->word);
  break;

https://github.com/mfragkoulis/bash/blob/master/parse.y#L6030
if (tok != WORD && tok != ASSIGNMENT_WORD)


-- 
Regards,
Peng

Re: What is the purpose of wdcache and wlcache?

[Peng Yu]

On Fri, Feb 8, 2019 at 10:50 AM Chet Ramey  wrote:
>
> On 2/8/19 10:52 AM, Peng Yu wrote:
> > On Fri, Feb 8, 2019 at 9:42 AM Chet Ramey  wrote:
> >>
> >> On 2/8/19 10:39 AM, Peng Yu wrote:
> >>>> Yes: ocache_free.
> >>>
> >>> Could you please help explain what wdcache and wlcache actually do.
> >>> Why is it essential to have them? Why not just alloc and free them
> >>> without the caches? Thanks.
> >>
> >> To avoid potentially-expensive calls to malloc and free, the same as
> >> any cache.
> >
> > There are already many mallocs and frees used in other places in the
> > program, why it is essential to use cache here. Is this decision based
> > on profiling or it is just based on some abstract concept but without
> > actual runtime data?
>
> It was based on profiling at the time I implemented it.
>
> >
> > Where are these parameters coming from?
> >
> > #define WDCACHESIZE 128
>
> That's a reasonable size for a cache, and it provided a nice balance
> between memory use and speed for the most common cases.
>
> > if ((nbytes) <= 32) {
>
> That's the byte threshold for this implementation of Duff's Device. If
> the object is larger than that, it's better to use memset, with whatever
> assist the compiler can give you.
>
> > I made the following test program. I don't understand why a longer
> > word "xyzabc" can still use the cache. What is actually stored in
> > "data"?
>
> Come on. If you're not going to read the code, at least read the comments.

This is what I don't understand. Why caching pointers not caching the
actual memory allocated to the string can improve the performance?

> /* Create an object cache C of N pointers to OTYPE. */
>
> /* Free all cached items, which are pointers to OTYPE, in object cache C. */

-- 
Regards,
Peng

Re: What is the purpose of wdcache and wlcache?

[Peng Yu]

On Fri, Feb 8, 2019 at 9:42 AM Chet Ramey  wrote:
>
> On 2/8/19 10:39 AM, Peng Yu wrote:
> >> Yes: ocache_free.
> >
> > Could you please help explain what wdcache and wlcache actually do.
> > Why is it essential to have them? Why not just alloc and free them
> > without the caches? Thanks.
>
> To avoid potentially-expensive calls to malloc and free, the same as
> any cache.

There are already many mallocs and frees used in other places in the
program, why it is essential to use cache here. Is this decision based
on profiling or it is just based on some abstract concept but without
actual runtime data?

Where are these parameters coming from?

#define WDCACHESIZE 128
if ((nbytes) <= 32) {

I made the following test program. I don't understand why a longer
word "xyzabc" can still use the cache. What is actually stored in
"data"?

$ cat main.c
/* vim: set noexpandtab tabstop=2: */

/* A structure which represents a word. */
typedef struct word_desc {
char *word;   /* Zero terminated string. */
int flags;/* Flags associated with this word. */
} WORD_DESC;

WORD_DESC *make_word(const char*);
void dispose_word(WORD_DESC *w);

#include 

#define PTR_T char*

typedef struct objcache {
PTR_T data;
int cs;   /* cache size, number of objects */
int nc;   /* number of cache entries */
} sh_obj_cache_t;

extern sh_obj_cache_t wlcache;
extern sh_obj_cache_t wdcache;

void print_WORD_DESC(const WORD_DESC* w) {
printf("{ word: %s, flags: %d }\n", w->word, w->flags);
}

void print_sh_obj_cache_t(const sh_obj_cache_t cache) {
printf("{ data: %s, cs: %d, nc: %d }\n", cache.data, cache.cs, cache.nc);
}

void cmd_init(void);

int main(int argc, char **argv, char **env) {
cmd_init ();
WORD_DESC *w = make_word("abc");
print_WORD_DESC(w);
print_sh_obj_cache_t(wdcache);
dispose_word(w);
print_sh_obj_cache_t(wdcache);
WORD_DESC* w1 = make_word("abc");
print_WORD_DESC(w1);
print_sh_obj_cache_t(wdcache);
WORD_DESC* w2 = make_word("abc");
print_WORD_DESC(w2);
print_sh_obj_cache_t(wdcache);
dispose_word(w1);
print_sh_obj_cache_t(wdcache);
dispose_word(w2);
print_sh_obj_cache_t(wdcache);
WORD_DESC* w3 = make_word("xyzabc");
print_sh_obj_cache_t(wdcache);
}
$./main.exe
{ word: abc, flags: 0 }
{ data: 
,
cs: 128, nc: 0 }
{ data:?s, cs: 128, nc: 1 }
{ word: abc, flags: 0 }
{ data:?s, cs: 128, nc: 0 }
{ word: abc, flags: 0 }
{ data:?s, cs: 128, nc: 0 }
{ data:?s, cs: 128, nc: 1 }
{ data:?s, cs: 128, nc: 2 }
{ data:?s, cs: 128, nc: 1 }

-- 
Regards,
Peng

Re: What is the purpose of wdcache and wlcache?

[Peng Yu]

> Yes: ocache_free.

Could you please help explain what wdcache and wlcache actually do.
Why is it essential to have them? Why not just alloc and free them
without the caches? Thanks.

-- 
Regards,
Peng

What is the purpose of wdcache and wlcache?

[Peng Yu]

Hi,

I don't understand the purpose of wdcache and wlcache. The "nc" field
seems to be always 0 (as initialized in ocache_create(), and I don't
find where it is increased. But `ocache_alloc()` just call xmalloc
without using the cache since nc is 0. So wdcache and wlcache seem to
be useless.

Do I miss something?

http://git.savannah.gnu.org/cgit/bash.git/tree/make_cmd.c#n52
sh_obj_cache_t wdcache = {0, 0, 0};
sh_obj_cache_t wlcache = {0, 0, 0};

http://git.savannah.gnu.org/cgit/bash.git/tree/make_cmd.c#n64
void
cmd_init ()
{
  ocache_create (wdcache, WORD_DESC, WDCACHESIZE);
  ocache_create (wlcache, WORD_LIST, WLCACHESIZE);
}

http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n55
typedef struct objcache {
PTR_T data;
int cs; /* cache size, number of objects */
int nc; /* number of cache entries */
} sh_obj_cache_t;

http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n29
#  define PTR_T char *

http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n61
/* Create an object cache C of N pointers to OTYPE. */
#define ocache_create(c, otype, n) \
do { \
(c).data = xmalloc((n) * sizeof (otype *)); \
(c).cs = (n); \
(c).nc = 0; \
} while (0)

http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n90
#define ocache_alloc(c, otype, r) \
  do { \
if ((c).nc > 0) { \
  (r) = (otype *)((otype **)((c).data))[--(c).nc]; \
} else \
  (r) = (otype *)xmalloc (sizeof (otype)); \
  } while (0)

--
Regards,
Peng

Re: What is the purpose of parser-built?

[Peng Yu]

On Wed, Feb 6, 2019 at 4:49 PM Eric Blake  wrote:
>
> On 2/6/19 4:18 PM, Peng Yu wrote:
> > Hi,
> >
> > I deleted the file parser-built, and bash still compiles and an empty
> > parser-built file will be generated upon compilation. What is the
> > purpose of this file? Should it be deleted? Thanks.

> parser-built is a witness that $(YACC) was run, even if the timestamps
> did not change (because the generated file did not change compared to
> last time). It exists in the file system so as to let make compute
> timestamp dependencies where we know the parser is up-to-date, even
> though the actual file we depend on has a timestamp that does NOT change
> (because we intentionally don't override it when there is no
> difference), all in order to minimize the time spent rebuilding the
> project when making a tweak to parse.y (for example, being able to tell
> the difference between a minor edit that changes a comment but not the
> generated parser, vs. a major edit that requires rebuilding other files
> to pick up the changes implied by the changed parser).

I don't get the point. Should parser-built be deleted? If bash can be
compiled without it, why put it in the source tar.gz file?

-- 
Regards,
Peng

Where is yacc_EOF defined?

[Peng Yu]

Hi,

yacc_EOF is mentioned in parse.y in something like this

%left '&' ';' '\n' yacc_EOF
  | error yacc_EOF

But I don't find where it is defined similarly to other tokens like BAR_AND.

%token GREATER_BAR BAR_AND

Where is yacc_EOF defined?

(y.tab.c and y.tab.h are files generated by bison. so yacc_EOF must be
defined before bison is called?)

-- 
Regards,
Peng

What is the purpose of parser-built?

[Peng Yu]

Hi,

I deleted the file parser-built, and bash still compiles and an empty
parser-built file will be generated upon compilation. What is the
purpose of this file? Should it be deleted? Thanks.

-- 
Regards,
Peng

Re: The use of register keyword in bash source code

[Peng Yu]

> No, that is what volatile means.  The register keyword is just an
> optimisation hint, and is mostly ignored by the compiler.

If it is ignored anyway, why "register" is used in many places in the
code? Thanks.

-- 
Regards,
Peng

The use of register keyword in bash source code

[Peng Yu]

Hi,

I see many variables are declared with the "register" keyword. I know
its purpose is to tell compile always access the corresponding memory
without assuming the previously accessed values are preserved. This is
usually to deal with some external devices.

But I don't understand why it is useful in bash. Could anybody help
explain? Thanks.

-- 
Regards,
Peng

Re: Where is GLOBAL_COMMAND?

[Peng Yu]

> That's a documentation convention - the all-caps in the docstring calls
> your attention to the need to search case-insensitively for the actual
> variable, while spelling it case-sensitively would make it blend into
> the sentence and make it harder to realize that the sentence is indeed
> pointing you to an external variable.

Why not just use ` to quote it literally? Given that markdown is very
common nowadays. Quoting it with ` should be a better choice rather
than making it capitalized. I guess this probably because the original
codebase predates markdown and its variants?

> If you think the existing convention is confusing, then submit a patch
> to change it instead of asking someone else to take on the grunt work.
> Otherwise, learn to live with the existing convention (which doesn't
> bother me, so I won't be submitting a patch).

If it was hosted on github, then I would. But I am not used to
savannah. Personally, I don't think it is as convenient.

-- 
Regards,
Peng

Re: Where is GLOBAL_COMMAND?

[Peng Yu]

> grep global_command *.?

GLOBAL_COMMAND is uppercase. But the actual variable name
global_command is in lowercase.

I think that GLOBAL_COMMAND should be changed to global_command in the comment.

-- 
Regards,
Peng

Where is GLOBAL_COMMAND?

[Peng Yu]

Hi,

GLOBAL_COMMAND is mentioned as a global variable. But I don't find it.
Is it renamed to something else?

eval.c
276-/* Call the YACC-generated parser and return the status of the parse.
277-   Input is read from the current input stream (bash_input).  yyparse
278:   leaves the parsed command in the global variable GLOBAL_COMMAND.
279-   This is where PROMPT_COMMAND is executed. */
280-int
--
310-
311-/* Read and parse a command, returning the status of the parse.  The command
312:   is left in the globval variable GLOBAL_COMMAND for use by reader_loop.
313-   This is where the shell timeout code is executed. */
314-int

execute_cmd.c
373-
374-/* Execute the command passed in COMMAND.  COMMAND is exactly what
375:   read_command () places into GLOBAL_COMMAND.  See "command.h" for the
376-   details of the command structure.
377-
--
540-
541-/* Execute the command passed in COMMAND, perhaps doing it asynchronously.
542:   COMMAND is exactly what read_command () places into GLOBAL_COMMAND.
543-   ASYNCHROUNOUS, if non-zero, says to do this command in the background.
544-   PIPE_IN and PIPE_OUT are file descriptors saying where input comes

-- 
Regards,
Peng

loadable cat bug?

[Peng Yu]

When I use the loadable cat, I may get the following error. The input
is a fifo in this specific case.

cat: cannot open /tmp/tmp.VXkbqFlPtH: Interrupted system call

So far, I can not make a minimal script to demonstrate the problem.
But if I replace it with coreutils cat in my code, the problem is
gone.

Does anybody know what could cause this error?

-- 
Regards,
Peng

~60 global variables use by only one function in bash source

[Peng Yu]

Hi,

I see these global or static variables (1st column) used only by one
function (2nd column). Some are from bash, some are from the libraries
that bash depends.

It seems to be problematic to declare variables global/static but only
to use them in one function. Should these variables be made local?

Thanks.

BC_rl_init_terminal_io
PC_rl_init_terminal_io
UP_rl_init_terminal_io
_rl_color_indicator_rl_parse_colors
_rl_complete_mark_symlink_dirsset_completion_defaults
_rl_completion_case_mapcomplete_fncmp
_rl_echo_control_charsrl_echo_signal_char
_rl_enable_bracketed_pasterl_prep_terminal
_rl_enable_metareadline_internal_setup
_rl_interrupt_immediatelyrl_signal_handler
_rl_mark_modified_linesrl_redisplay
_rl_match_hidden_filesrl_filename_completion_function
_rl_menu_complete_prefix_firstrl_menu_complete
_rl_optimize_typeaheadrl_insert
_rl_page_completionsrl_display_match_list
_rl_print_completions_horizontallyrl_display_match_list
_rl_revert_all_at_newlinereadline_internal_teardown
_rl_skip_completed_textinsert_match
_rl_term_clrscroll_rl_init_terminal_io
_rl_term_pc_rl_init_terminal_io
alias_expand_allalias_expand
bash_copyrightshow_manpage
bash_getcwd_errstrget_working_directory
bash_licenseshow_manpage
build_versionmake_vers_array
current_command_subst_pidcommand_substitute
debugging_login_shellmain
dircomplete_expand_relpathbash_directory_completion_hook
dont_save_function_defsreset_shell_options
from_return_trapexecute_function
funmap_program_specific_entry_startrl_initialize_funmap
history_multiline_entriesread_history_range
history_no_expand_charshistory_expand
line_number_baseassign_lineno
localvar_unsetmakunbound
max_input_historystifle_history
optflagsinitialize_flags
pretty_print_modemain
progcomp_aliasprogrammable_completions
release_statusmake_vers_array
rl_already_promptedreadline_internal_setup
rl_change_environment_rl_get_screen_size
rl_complete_with_tilde_expansionrl_filename_completion_function
rl_completion_word_break_hook_rl_find_completion_word
rl_event_hookrl_read_key
rl_executing_key_rl_dispatch_subseq
rl_executing_keymap_rl_dispatch_subseq
rl_inhibit_completionrl_complete
rl_input_available_hook_rl_input_available
rl_pre_input_hookreadline_internal_setup
rl_prefer_env_winsize_rl_get_screen_size
rl_visible_statsprint_filename
saved_command_line_countedit_and_execute_command
sh_syntabsizlocale_setblanks
shell_script_filenamemain
sourcenest_maxexecute_builtin
terminate_immediatelytermsig_sighandler
the_printed_command_sizethe_printed_command_resize
this_shell_functionexecute_function
tilde_expansion_failure_hooktilde_expand_word
vi_insertion_keymaprl_vi_replace
yylvalyyparse
yynerrsyyparse

-- 
Regards,
Peng

Re: Difference of extglob between 5.0.0(1)-release and 4.4.23(1)-release

[Peng Yu]

> The bash-4.4 code only worked the way you want it by chance. There was a
bug that was fixed in January, 2017, the result of

> http://lists.gnu.org/archive/html/bug-bash/2017-01/msg00018.html

> that uncovered the behavior you're complaining about.

This only explains where the change of behavior for my example comes
from. I assume that this is what you meant and you did not mean to use
that email to justify the change for the example that I showed.

> In a filename context, or a context where a leading `.' must be matched
> explicitly, a pattern must only match a filename that starts with a `.' if
> `.' is the first character in the pattern. A pattern that begins with a
> null extglob pattern (especially one that is defined to perform at least
> one match) followed by a dot, quoted or unquoted, does not fulfill that
> criterion.

I don't follow your explanation. Could you please use my specific
examples and break it down into steps to show what globbing pattern
matching is involved in bash5? And why the logic in bash4 was wrong?

> I wouldn't be so quick to declare this a bug. Other shells (e.g, ksh93,
> mksh, and zsh) that implement extended globbing patterns behave like
> bash-5.0 does.
>
> There is a question of whether or not an extglob pattern that is allowed
> to make zero matches followed by a `.' should succeed, and the existing
> implementations are mixed on that point.

I don't think it is a good idea to introduce such kind of special
cases. If @() should match an empty string, the least surprising
definition is that it should match empty string everywhere. Weight the
surprise that it could introduce, does the benefit of introducing
special cases large enough to allow those special cases?

-- 
Regards,
Peng

Re: Identical function names in bash source code

[Peng Yu]

>
>
> > https://pastebin.com/cV1jP41Y
>
> Really? What is your analysis? There are 100 duplicate global symbols
> shared between bash and other libraries? Or is it your assertion that
> one should never use the same symbol names, unconditionally? You're not
> making much of a point here.


It is the duplicated definitions among all the files in the source
directory, not just the ones in the package root directory versus the ones
below the root directory. The first column is the symbol, the second column
is the files where a symbol is defined. You can check some examples to see
if the analysis result makes sense.

>
> --
Regards,
Peng

Re: Identical function names in bash source code

[Peng Yu]

> "Not uncommon" is stretching it, since it happens in only one place:
> lib/readline/shell.c.

No. it is not uncommon. See the analysis of duplicated function/macro
names and where they appear. There are around ~100 of them. Note that
this analysis is not very accurate. But the balkpark estimate should
be OK.

https://pastebin.com/cV1jP41Y

> Readline is now distributed separately, and used in hundreds of other
> applications in addition to bash, but it was not always this way.

Then, since it has been made into a separate package long time ago,
why readline has not been removed from bash source code?

Thanks.

-- 
Regards,
Peng

Re: [Help-bash] What are the regex spec for function names?

[Peng Yu]

> There is probably no easy regex to match strings bash will tolerate as
> a function name without error. The accepted names vary in several
> contexts.
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05
>
> "The function is named fname; the application shall ensure that it is
> a name (see XBD Name) and that it is not the name of a special
> built-in utility. An implementation may allow other characters in a
> function name as an extension."
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_235
>
> So:
>
> ^[_[:alpha:]][_[:alnum:]]+$
>
> is close enough for most purposes. There is no easy regex to filter
> the set of special builtin names.

Could anybody help explain How this is done at the parsing?

I am still learning the code. It seems that bash does not rely on flex
to tokenize. But how to recognize function that can not be expressed
in regex?

-- 
Regards,
Peng

Re: Identical function names in bash source code

[Peng Yu]

> What would you say the "suggested improvement" is here?

This is implied. If it is agreed that identical function names are not
good by the majority of bash developers, then what I found could be
turned into an explicit suggestion.

Since maybe there is a good reason, I don't want to pretend that I
knew why it was designed in this way.

So far I don't see a good reason. So I want to confirm it first.

Or this problem has never been noted before (I don't think it is very
likely, given bash has been been developed for a long time). Then, it
worth a discussion.

To me, help-bash is more for using bash for scripting. bug-bash is
more appropriate for bash development questions.

-- 
Regards,
Peng

Identical function names in bash source code

[Peng Yu]

Hi,

It is not uncommon to see the same name is used to defined functions
in different .c files in bash source code.

For example, sh_single_quote is defined in both lib/readline/shell.c
and lib/sh/shquote.c with the exact same signature. The two pieces of
code are slightly different. Do they do the exact same things or do
something different?

In either case, is having the same name for different functions a good
practice? This will make the linked binary dependent on the order of
the corresponding .a files specified. Or if linked via .o files, then
one function will shadow the others. See 1) and 2) below for minimal
working examples. Neither cases seem to be good and could be avoided
easily by giving the functions unique names.

So, should such functions with the same name be named differently? Thanks.

// lib/readline/shell.c
/* Does shell-like quoting using single quotes. */
char *
sh_single_quote (char *string)
{
  register int c;
  char *result, *r, *s;

  result = (char *)xmalloc (3 + (4 * strlen (string)));
  r = result;
  *r++ = '\'';

  for (s = string; s && (c = *s); s++)
{
  *r++ = c;

  if (c == '\'')
  {
*r++ = '\\';  /* insert escaped single quote */
*r++ = '\'';
*r++ = '\'';  /* start new quoted string */
  }
}

  *r++ = '\'';
  *r = '\0';

  return (result);
}

// lib/sh/shquote.c
/* Return a new string which is the single-quoted version of STRING.
   Used by alias and trap, among others. */
char *
sh_single_quote (string)
 const char *string;
{
  register int c;
  char *result, *r;
  const char *s;

  result = (char *)xmalloc (3 + (4 * strlen (string)));
  r = result;

  if (string[0] == '\'' && string[1] == 0)
{
  *r++ = '\\';
  *r++ = '\'';
  *r++ = 0;
  return result;
}

  *r++ = '\'';

  for (s = string; s && (c = *s); s++)
{
  *r++ = c;

  if (c == '\'')
  {
*r++ = '\\';  /* insert escaped single quote */
*r++ = '\'';
*r++ = '\'';  /* start new quoted string */
  }
}

  *r++ = '\'';
  *r = '\0';

  return (result);
}

# 1 ##
==> libprint1.c <==
// vim: set noexpandtab tabstop=2:
#include 

void print() {
  puts("Hello World1!\n");
}

==> libprint2.c <==
// vim: set noexpandtab tabstop=2:
#include 

void print() {
  puts("Hello World2!\n");
}

==> main.c <==
// vim: set noexpandtab tabstop=2:

void print();
int main() {
  print();
  return 0;
}

==> main.sh <==
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint1.a libprint1.o
ar cr libprint2.a libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint1.a libprint2.a
./main.exe
gcc -o main.exe main.o libprint2.a libprint1.a
./main.exe
$ ./main.sh
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint1.a libprint1.o
ar cr libprint2.a libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint1.a libprint2.a
./main.exe
Hello World1!

gcc -o main.exe main.o libprint2.a libprint1.a
./main.exe
Hello World2!

# 2 ##
==> libprint1.c <==
// vim: set noexpandtab tabstop=2:
#include 

void print() {
  puts("Hello World1!\n");
}

==> libprint2.c <==
// vim: set noexpandtab tabstop=2:
#include 

void print() {
  puts("Hello World2!\n");
}

==> main.c <==
// vim: set noexpandtab tabstop=2:

void print();
int main() {
  print();
  return 0;
}

==> main.sh <==
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint.a libprint1.o libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint.a
./main.exe
ar cr libprint.a libprint2.o libprint1.o
gcc -o main.exe main.o libprint.a
./main.exe

$ ./main.sh
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint.a libprint1.o libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint.a
./main.exe
Hello World1!

ar cr libprint.a libprint2.o libprint1.o
gcc -o main.exe main.o libprint.a
./main.exe
Hello World1!

-- 
Regards,
Peng

JIT in bash

[Peng Yu]

The following test cases show that the variable length can
significantly affect the runtime in bash. But the variable length
doesn't seem to have a significant effect in some other interpreted
languages, such as python.

I can understand that variables length will slow down the runtime, as
bash is interpreted. But I didn't expect that it should slow down the
runtime so much. Does anybody happen to know the cause of the problem
on why bash is slowed down so much compared with python when the
variable length is increased?

One way that I think can mitigate (but not solve the original problem)
the problem is to use just-in-time (JIT) compilation.

Python has a JIT compiler pypy. However, I don't see that bash has a
JIT compiler. Is it because people think that bash is just a glue
language so that making it run faster is not important, therefore
there is no need to make a JIT compiler for bash?

Or it is because the current bash code started very early and has
certain limitation which makes it hard to create a new JIT compiler
based on it?

$ cat ./main.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

TIMEFORMAT=%R
set -v
time for ((i=0;i<1;++i)); do
:
done > /dev/null

time for 
((i=0;i<1;++i));
do
:
done > /dev/null

time ./main.py 100
time ./main.py 1000
time ./main_long.py 100
time ./main_long.py 1000
$ cat main.py
#!/usr/bin/env python
# vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8:

import sys
for i in xrange(int(sys.argv[1])):
pass
$ ./main.sh
time for ((i=0;i<1;++i)); do
:
done > /dev/null
0.148

time for 
((i=0;i<1;++i));
do
:
done > /dev/null
0.321

time ./main.py 100
0.188
time ./main.py 1000
0.822
time ./main_long.py 100
0.164
time ./main_long.py 1000
0.945


-- 
Regards,
Peng

Re: [Help-bash] How to run tests/?

[Peng Yu]

> Have you tried 'make test'?

No, I didn't. I didn't know it was a target. I just followed the
README in that directory.

-- 
Regards,
Peng

Re: bug in dirname loadable?

[Peng Yu]

On Wed, Dec 26, 2018 at 11:35 AM Chet Ramey  wrote:
>
> On 12/24/18 10:35 PM, Peng Yu wrote:
> > dirname loadable gives the following error. I think the coreutils'
> > direname's convention is better. Should it be considered as a bug to
> > fix?
> >
> > $ dirname -- -a
> > dirname: usage: dirname string
> > $(type -P dirname) -- -a
> > .
>
> Yes, dirname should skip over a `--' denoting the end of options. Thanks
> for the report.

There is a similar problem in `basename`. Is it due to a common bug on
how loadables are programmed?

$ builtin basename -a
-bash: basename: -a: invalid option
basename: usage: basename string [suffix]
$ builtin basename -- -a
--

-- 
Regards,
Peng

The usage of `cd builtins && $(MAKE) ...`

[Peng Yu]

Hi,

I see things like `cd builtins && $(MAKE) ...` in the Makefiles in
bash source code. GNU Make has the option of -C for entering a
directory and make. Is the reason to cd then make for compatibility
with other make's that don't support -C? Thanks.

-- 
Regards,
Peng

Re: How to compile hashlib.c for testing?

[Peng Yu]

> That code hasn't really changed in almost twenty years. All the testing
> was done long ago.

Do you keep all the testing code in the bash repository? Or you keep
the testing code separately from the bash source? Given the fugal
testing code that is in the bash source, it is doesn't seem that the
current testing code can cover all the branches of the source code. I
am wondering how to ensure the code contains as few bugs as possible
without full coverage testing cases? Thanks.

-- 
Regards,
Peng

Re: Should [[ -v 1 ]] be supported?

[Peng Yu]

> A profiler is exactly what you need here. You should profile your
> script and understand the stuff that actually matters for your goals.
> Otherwise you're just chasing unimportant things.

Again, my goal is not to profile a specific bash script. The goal is
to see what features make bash only fit into a shell language but
cannot make into other domains that other languages (e.g. python) are
popular at.

-- 
Regards,
Peng

Re: Should [[ -v 1 ]] be supported?

[Peng Yu]

> You're whacking moles.  Use a profiler.  That's what they're for.

I've already shown that $() is a major problem to slow down the speed
and I have reduced using its usage in my code and significantly
improved the performance. Nevertheless, it doesn't mean that it is not
necessary to systematically evaluating features that are used
frequently.

> Yes, I can.  You need to identify where bash is _actually_ spending most
> of its execution time, and a profiler can help you do that.

Yes and no. For a particular bash script, you can quantify which bash
features are the most time-consuming. But you can not profile all the
bash scripts that have ever been written. Since there are only limited
features in bash, in this case, a logical way to go is to at least
profile each commonly used feature with minimal code (as just for and
repetitive calling that features as I do) and understand its pros and
cons. A profiler is an overkill in this case.

-- 
Regards,
Peng

Re: How to compile hashlib.c for testing?

[Peng Yu]

We are talking about unit testing in the bash C source code, not bash scripts.

On Thu, Dec 27, 2018 at 8:03 PM G. Branden Robinson
 wrote:
>
> At 2018-12-27T17:34:49-0800, Eduardo Bustamante wrote:
> > On Thu, Dec 27, 2018 at 5:15 PM Peng Yu  wrote:
> > (...)
> > > Since the main() function is already there, why there is not already
> > > an easy way to compile it? How do you do unit-testing then for the
> > > code?
> >
> > This is very easy to figure out from the source code, right :)?
> >
> > (Hint: there is no "unit" testing in bash)
>
> Au contraire!  :-D
>
> https://github.com/kward/shunit2
>
> Regards,
> Branden



-- 
Regards,
Peng

Re: Should [[ -v 1 ]] be supported?

[Peng Yu]

On Thu, Dec 27, 2018 at 7:37 PM G. Branden Robinson
 wrote:
>
> At 2018-12-27T18:39:26-0600, Peng Yu wrote:
> > What I meant in my original email is that I want something for testing
> > if there is a command line argument (one or more, the exact number
> > does not matter). $# gives more than that info, because it tells not
> > only whether is any command line argument, but also how many. This
> > could lead to slower performance if the goal is to just test if there
> > is an argument.
>
> You should look into how integer comparisons are done in hardware.
>
> For instance, comparison and subtraction operations are often comparable
> (or even identical) in cycle count because they both perform a
> subtraction "under the hood".  You need to be programming in assembly
> language to influence execution at such depths.
>
> See, e.g.,
>
> https://www.quora.com/What-is-the-difference-between-cmp-and-sub-instruction-in-8086microprocessor
>
> As others have noted, if you are worried about marginal performance
> impacts this small, margin you are probably writing in the wrong
> language, or distracting yourself with tiny details when you do not even
> know the cyclomatic complexity of your code or the big-O classification
> of your algorithms.
>
> Attack those problems first, and see what you discover.

The problem is that bash is not systematically profiled for
performance at all. I am doing it step it by step. There are many more
that I have already tested. You can not dismiss this one just because
it may not have a large impact.

-- 
Regards,
Peng

Re: Should [[ -v 1 ]] be supported?

[Peng Yu]

> I don't believe that at all. The number of positional parameters is kept
> anyway. It's not recalculated when you compare it to another number, so
> it's just as fast as a simple comparison of two integers.

Getting the number $# is slow.

> And even if it weren't -- if performance is *that* important to you,
> you're using the wrong language altogether.

There is no reason why bash has to be slow. Javascript was slow. But
it has been optimized to be much faster than it was before.

What I see bash is that it is syntax is much better than other
languages in terms of interactive use and not use () for arguments? It
could be made faster and with some additional features, it could be
used for other more serious applications rather than just a glue of
other tools.

-- 
Regards,
Peng

Re: How to compile hashlib.c for testing?

[Peng Yu]

On Thu, Dec 27, 2018 at 12:27 PM Chet Ramey  wrote:
>
> On 12/26/18 4:31 PM, Peng Yu wrote:
> > Hi,
> >
> > I'd like to compile hashlib.c to try its main(). But I got the
> > following error. What is the correct commands to compile it? Thanks.
>
> Think about the error message:
>
> >   "_xmalloc", referenced from:
> >   _hash_create in hashlib.o
> >   _hash_copy in hashlib.o
> >   _hash_search in hashlib.o
> >   _hash_insert in hashlib.o
> >   _main in hashlib.o
> > ld: symbol(s) not found for architecture x86_64
>
> So you need a definition for xmalloc. The easiest thing to do is to add
> an xmalloc function in the TEST_HASHING block, or you could follow the
> directions in the comment there.
>
> Sooner or later, you're going to have to pick this stuff up on your own.


I run make in lib/malloc, then I got the following error. What is wrong?

~/Downloads/bash-4.4$ gcc hashlib.o xmalloc.o lib/malloc/libmalloc.a
-o hashlib.exe
Undefined symbols for architecture x86_64:
  "_libintl_gettext", referenced from:
  _xmalloc in xmalloc.o
  _xrealloc in xmalloc.o
  _internal_malloc in libmalloc.a(malloc.o)
  _internal_realloc in libmalloc.a(malloc.o)
  _internal_free in libmalloc.a(malloc.o)
  _xbotch in libmalloc.a(malloc.o)
  "_running_trap", referenced from:
  _internal_malloc in libmalloc.a(malloc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Then I add ./lib/intl/libintl.a, I got additional errors. This does
not seem to be an efficient way to resolve the problem. Is there an
easy way to figure out all the necessary libraries (but no more) to
link.

$ gcc hashlib.o  xmalloc.o ./lib/intl/libintl.a lib/malloc/libmalloc.a
-o hashlib.exe
Undefined symbols for architecture x86_64:
  "_iconv", referenced from:
  __nl_find_msg in libintl.a(dcigettext.o)
  "_iconv_close", referenced from:
  __nl_free_domain_conv in libintl.a(loadmsgcat.o)
  "_iconv_open", referenced from:
  __nl_init_domain_conv in libintl.a(loadmsgcat.o)
  "_libiconv_set_relocation_prefix", referenced from:
  _libintl_set_relocation_prefix in libintl.a(relocatable.o)
  "_running_trap", referenced from:
  _internal_malloc in libmalloc.a(malloc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Since the main() function is already there, why there is not already
an easy way to compile it? How do you do unit-testing then for the
code?

-- 
Regards,
Peng

Re: Should [[ -v 1 ]] be supported?

[Peng Yu]

On Thu, Dec 27, 2018 at 3:19 PM Martijn Dekker  wrote:
>
> Op 27-12-18 om 19:22 schreef Chet Ramey:
> > On 12/26/18 10:49 PM, Peng Yu wrote:
> >
> >> Although [[ -z ${1+s} ]]  and (($#)) works for testing if $1 is set,
> >> neither of them are uniformly better performance wise. In this case,
> >> should [[ -v 1 ]] be supported?
> >
> > So you're saying that neither of the existing options performs better
> > than the other, though they both perform well, so we should add some
> > new capability just because? That's a particularly poor argument.
>
> Consistency might be a better argument. If [[ -v foo ]] is equivalent to
> [[ -n ${foo+s} ]] for variables (with the advantage that you don't need
> 'eval' to handle arbitrary values of 'foo'), then perhaps it's not
> unreasonable to expect [[ -v 1 ]] to be equivalent to [[ -n ${1+s} ]].
>
> FWIW, zsh and mksh do support this; ksh93 doesn't.

The above are additions arguments that why [[ -v 1 ]] should be supported.

What I meant in my original email is that I want something for testing
if there is a command line argument (one or more, the exact number
does not matter). $# gives more than that info, because it tells not
only whether is any command line argument, but also how many. This
could lead to slower performance if the goal is to just test if there
is an argument.

[[ -z ${1+s} ]] does something also more than necessary too, because
it not only tests for whether $1 is set, it also replaced with a
string "s". This also does more than just testing whether $1 is set.

So both cases would be slower than [[ -v 1 ]] if it were supported.

As of now, because (($#)) or [[ -z ${1+s} ]] are not consistently
faster than the other, there is no way to write a program that is
consistently fastest. To achieve this goal, one has to implement [[ -v
1 ]] or something similar that just test whether $1 but no more.

-- 
Regards,
Peng

Should [[ -v 1 ]] be supported?

[Peng Yu]

Hi,

[[ -v 1 ]] does not work for $1.

$ [ -v 1 ]; echo "$?"
1
$ set -- a
$ [ -v 1 ]; echo "$?"
1

Although [[ -z ${1+s} ]]  and (($#)) works for testing if $1 is set,
neither of them are uniformly better performance wise. In this case,
should [[ -v 1 ]] be supported?

set -- $(seq 1)
time for ((i=0;i<1;++i)); do
(($#))
done

real0m0.140s
user0m0.132s
sys0m0.001s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done

real0m0.216s
user0m0.151s
sys0m0.000s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done

real0m0.336s
user0m0.156s
sys0m0.009s

set -- $(seq 10)
time for ((i=0;i<1;++i)); do
(($#))
done

real0m0.224s
user0m0.130s
sys0m0.001s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done

real0m0.256s
user0m0.147s
sys0m0.000s

set -- $(seq 100)
time for ((i=0;i<1;++i)); do
(($#))
done

real0m0.260s
user0m0.131s
sys0m0.001s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done

real0m0.244s
user0m0.146s
sys0m0.001s

set -- $(seq 1000)
time for ((i=0;i<1;++i)); do
(($#))
done

real0m0.338s
user0m0.167s
sys0m0.002s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done

real0m0.277s
user0m0.144s
sys0m0.000s

set -- $(seq 1)
time for ((i=0;i<1;++i)); do
(($#))
done

real0m1.012s
user0m0.540s
sys0m0.011s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done

real0m0.292s
user0m0.148s
sys0m0.003s




-- 
Regards,
Peng

How to compile hashlib.c for testing?

[Peng Yu]

Hi,

I'd like to compile hashlib.c to try its main(). But I got the
following error. What is the correct commands to compile it? Thanks.

$ gcc  -DPROGRAM='"bash"' -DCONF_HOSTTYPE='"x86_64"'
-DCONF_OSTYPE='"darwin17.7.0"'
-DCONF_MACHTYPE='"x86_64-apple-darwin17.7.0"' -DCONF_VENDOR='"apple"'
-DLOCALEDIR='"/usr/local/share/locale"' -DPACKAGE='"bash"' -DSHELL
-DTEST_HASHING -DHAVE_CONFIG_H -DMACOSX   -I.  -I. -I./include -I./lib
-I./lib/intl -I/Users/pengy/Downloads/bash-4.4/lib/intl  -g -O2
-Wno-parentheses -Wno-format-security -c hashlib.c
$ gcc -c hashlib.o  hashlib.exe
Undefined symbols for architecture x86_64:
  "_xmalloc", referenced from:
  _hash_create in hashlib.o
  _hash_copy in hashlib.o
  _hash_search in hashlib.o
  _hash_insert in hashlib.o
  _main in hashlib.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

-- 
Regards,
Peng

difference between /tmp and other directory for loadable mkdir?

[Peng Yu]

Hi,

I can not mkdir -p . in /tmp/ via the loadable mkdir. What is the
difference between /tmp/ and other directories? I am on Mac OS X. Is
this a bug in mkdir?

$ cd /tmp
$ mkdir -p -- .
-bash: mkdir: .: Operation not permitted
$ cd ~/
$ mkdir -p -- .

-- 
Regards,
Peng

Re: Why does ctrl-c behave differently for builtin sleep in a subshell?

[Peng Yu]

If I add a command after the builtin sleep, then the EXIT trap will be
triggered upon typing ctrl-C.

If  the last command is removed, then the EXIT trap will not be
triggered upon typing ctrl-C.

Is this a bug in the built-in sleep? I am trying to inspect the C code
of sleep builtin, but it is very simple. I am not sure where the
problem comes from.

Could anybody take a look? Thanks.

$ cat main_INT.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

source ../../../trapexit.sh
(
trapexit -q <<-'EOF'
echo Hello World!
EOF
source enable.sh
enable sleep
sleep 10
echo "$?"
)
$   ./main_INT.sh
^CHello World!

$ cat main_INT.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

source ../../../trapexit.sh
(
trapexit -q <<-'EOF'
echo Hello World!
EOF
source enable.sh
enable sleep
sleep 10
#echo "$?"
)
$ ./main_INT.sh
^C
$ source enable.sh
$ enable sleep
$ help sleep
sleep: sleep seconds[.fraction]
Suspend execution for specified period.
sleep suspends execution for a minimum of SECONDS[.FRACTION] seconds.

On Mon, Dec 17, 2018 at 1:57 PM Peng Yu  wrote:
>
> Note that when SIGINT is set, whether an external or built-in sleep is
> used will not result in any difference.
>
> If I use some other method (say a for loop) instead of built-in sleep
> to introduce a delay, the EXIT can be reached upon ctrl-c. So this
> sounds like that the builtin sleep somehow causes the problem. Is it
> the case?
>
> $ cat main_builtin_INT.sh
> #!/usr/bin/env bash
> trap 'echo at EXIT' EXIT
> trap 'echo at SIGINT' SIGINT
> enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> sleep 10
> $ cat  \(/main_builtin_INT.sh
> #!/usr/bin/env bash
> (
> trap 'echo at EXIT' EXIT
> enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> sleep 10
> )
> $ ./main_builtin_INT.sh
> ^Cat SIGINT
> at EXIT
> $ '('/main_builtin_INT.sh
> ^Cat SIGINT
> at EXIT
>
> $ cat ./main_builtin_for.sh
> #!/usr/bin/env bash
> (
> trap 'echo at EXIT' EXIT
>     for ((i=0;i<10;++i)); do
> :
> done
> )
> $  ./main_builtin_for.sh
> ^Cat EXIT
>
> On Mon, Dec 17, 2018 at 1:41 PM Peng Yu  wrote:
> >
> > Hi,
> >
> > See the follow code, '(/main_builtin.sh' behaves differently from the
> > other three scripts upon receiving ctrl-c. This is counter-intuitive.
> > I'd expect whether a built-in command called the result should be the
> > same.
> >
> > How to understand why there is such a difference? Is there an easy
> > workaround to make the result the same as the other three?
> >
> > $   ./main.sh
> > ^Cat EXIT
> >
> > $   ./main_builtin.sh
> > ^Cat EXIT
> >
> > $ \(/main.sh
> > ^Cat EXIT
> >
> > $ \(/main_builtin.sh
> > ^C
> >
> > $ cat main.sh
> > #!/usr/bin/env bash
> > trap 'echo at EXIT' EXIT
> > sleep 10
> >
> > $ cat main_builtin.sh
> > #!/usr/bin/env bash
> > trap 'echo at EXIT' EXIT
> > enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> > sleep 10
> >
> > $ cat '(/main.sh'
> > #!/usr/bin/env bash
> > (
> > trap 'echo at EXIT' EXIT
> > sleep 10
> > )
> >
> > $ cat '(/main_builtin.sh'
> > #!/usr/bin/env bash
> > (
> > trap 'echo at EXIT' EXIT
> > enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> > sleep 10
> > )
> >
> > --
> > Regards,
> > Peng
>
>
>
> --
> Regards,
> Peng



-- 
Regards,
Peng

bug in dirname loadable?

[Peng Yu]

dirname loadable gives the following error. I think the coreutils'
direname's convention is better. Should it be considered as a bug to
fix?

$ dirname -- -a
dirname: usage: dirname string
$(type -P dirname) -- -a
.

-- 
Regards,
Peng

Re: general loadable integration

[Peng Yu]

> I suppose the first thing needed to make that work, and maybe the only
> thing needed to make that work, is agreement on the name of a search path
> environment variable that enable can use to find loadable builtins.

Why not just use an environment variable such as LOADABLES_PATH (just
like the way how PATH is used)? Is it too difficult to reach an
agreement on this? As of 4.4.23, I still don't see such an environment
variable. Anyway, any variable is better than no variable.

-- 
Regards,
Peng

Re: Why does bash use xmalloc?

[Peng Yu]

The artificial ulimit is to tigger the error.

My point is why bash terminates when it runs an external command that
requires a large memory. Shouldn't bash return an exit code on behalf
of the failed command and continue to the next command?

On Sunday, November 6, 2016, Eduardo Bustamante  wrote:

> Hi Peng. Read the link you provided again. xmalloc is not an
> alternative version of malloc. It's just a common wrapper function
> around malloc. You can go and see for yourself, the definition is
> here: http://git.savannah.gnu.org/cgit/bash.git/tree/xmalloc.c#n97
>
> If you want the rest of the commands execute properly, then make sure
> that there's enough memory in the system, and remove artificial
> restrictions (ulimit -v?). There's no trick here, and certainly no bug
> in bash.
>


-- 
Regards,
Peng

Re: How bash do tokenization?

[Peng Yu]

OK. I see it, which check emails and print prompt.

What factors people need to consider to decide whether to use flex to
perform tokenization and write a customize tokenizer?

Checking emails and printing prompt strictly speaking is not related
with tokenization. Is there an alternative way to organize the code so
that it will be more decoupled?

On Tue, Oct 18, 2016 at 10:22 PM, Eduardo Bustamante  wrote:
> On Tue, Oct 18, 2016 at 10:18 PM, Eduardo Bustamante  
> wrote:
>> Check parser.y
> Sorry, I meant parse.y, inside it you will find read_token and yylex.



-- 
Regards,
Peng

Re: [Help-bash] make function local

[Peng Yu]

Hi Chet,

 That's the difference: if you're careful with naming and rigorous about
 your calling conventions, your one-time-use functions are about as close
 as you can get to local functions in bash, but you have to pay attention
 to the declaration's side effects.

 There is at least a runtime overhead for having too many unused
 functions that are supposed to be local or in your word lambda.

 Maybe so, but this is not the discussion we were having.

 Despite that one can rename supposedly internal functions to names
 that are unlikely to cause name collision via good naming convention,
 it still can incur a significant performance overhead.

 I disagree that performance overhead in typical use is `significant'.
 This point is more or less identical to the one I discussed Friday in
 regards to creating huge numbers of variables.  Who creates 100,000
 shell functions in a single script?  The overhead, such as it is, of
 creating and calling even 10,000 functions is negligible.

Why do you assume that one only calls each of these functions once?
What if a function is called thousands of time?

The performance difference is 10x different (0.490s vs 0.040s). To me,
this is a huge performance problem.

 (And the example you chose to illustrate this is not what Linda is
 talking about or using.)

 In this sense, I think that it is still necessary to consider make the
 supposedly internal function local so that they would not slow down
 function search in the global namespace.

 Look, you can make the same argument about function creation at any
 scope, since there is one function namespace.

Functions are usually called for more times they are defined.
Therefore, even if the function creation time is acceptable, it
doesn't mean its call time is acceptable.

 Acceptable performance is
 subjective: if the technique that Linda uses for data encapsulation
 results in performance that's acceptable for her application, then it's
 ok for her to use it.  You are certainly free to use any methodology you
 find comfortable and satisfies your constraints.

The main point that Linda and I are making is that we suggest add the
local function feature. I believe that we've made the case --- it
makes the code cleaner and run faster.

What makes you reluctant to consider this feature be added? Is it
takes too much time to implement such a feature?

-- 
Regards,
Peng

Re: [Help-bash] make function local

[Peng Yu]

Hi Chet,

 Eduardo A. Bustamante López wrote:
 Well, if your scripts are so simple, why use local functions at all?
 ---
 Cleanliness, Hygiene...

 Please, let's not have this argument again.  I think you're all using the
 term `local function' to mean different things.

 You seem to be using the term to describe one-time-use functions (they're
 almost lambdas, but they have names).  If you define a function within
 another function's body, have it unset itself when it executes, and call
 it only from within the function where it's defined, you have something
 very close to one-time-use functions with function-only scope.

 However you use them, they share the name namespace as every other defined
 function, regardless of whether or not they are defined as part of the body
 of another function.  If you had a function defined in the global scope
 with the same name as your one-time-use function, it would be removed when
 the one-time-use function was declared.  I think this is what Greg and
 Eduardo mean, and in this sense they are correct: bash doesn't have local
 functions with separate namespaces from other defined functions.

 That's the difference: if you're careful with naming and rigorous about
 your calling conventions, your one-time-use functions are about as close
 as you can get to local functions in bash, but you have to pay attention
 to the declaration's side effects.

There is at least a runtime overhead for having too many unused
functions that are supposed to be local or in your word lambda.
Despite that one can rename supposedly internal functions to names
that are unlikely to cause name collision via good naming convention,
it still can incur a significant performance overhead.

In this sense, I think that it is still necessary to consider make the
supposedly internal function local so that they would not slow down
function search in the global namespace.

~$ cat main_many.sh
#!/usr/bin/env bash

n=$1
for i in $(seq -w $n)
do
  eval function f$i { echo '$i'; }
done

function g {
for i in $(seq -w $n | head -n 1000)
do
  f$i  /dev/null
done
}

time g

~$ ./main_many.sh 1000

real 0m0.032s
user 0m0.020s
sys 0m0.010s
~$ ./main_many.sh 1

real 0m0.040s
user 0m0.029s
sys 0m0.011s
~$ ./main_many.sh 10

real 0m0.490s
user 0m0.461s
sys 0m0.031s

-- 
Regards,
Peng

Re: How to deal with errors in ()?

[Peng Yu]

On Sat, Mar 14, 2015 at 8:46 AM, Linda Walsh b...@tlinx.org wrote:


 Peng Yu wrote:

 Hi,

 http://mywiki.wooledge.org/ProcessSubstitution

 The above webpage says the following.

 commandA (commandB; [commandB's exit code is available here from $?])
 [commandB's exit code cannot be obtained from here.  $? holds
 commandA's exit code]



 Does anybody have a good solution for this situation? Thanks.

 --
 It's not a pretty solution, but how about some variation of:

 alias cmda='cat /dev/tty'## 1st cmd
 alias cmdb='(echo foo;exit 2)'  ## command executes, but exits w/err=2
 read status  (cmda (cmdb; echo status=$? 2) 21)

Stdout and stderr are commonly used for other purposes. But using fd 3
and above seems to be a good walkaround. Does anybody have a cleaner
solution?

 status=2
 foo
 ---
 not ideal, but it does get you the status.
 Instead of echoing it you could assigned it to a var,
 and test the value and only print status if non-zero,
 something like:

   ((status=$?))  echo status=$status

 instead of just the echo





-- 
Regards,
Peng

Re: -i option of set missing in man bash

[Peng Yu]

On Friday, March 13, 2015, Chet Ramey chet.ra...@case.edu wrote:

 On 3/12/15 2:13 PM, Peng Yu wrote:
  The -i option obviously works with set. But it is missing in the man
  page. Should this be added?
 
  No.  It's really only there for completeness, so things like `set $-'
  work as expected without error.
 
  But if something is in the implementation, it should be also in the
  documentation, right?

 There is a level of absurdity about this.  I can put a sentence in saying
 that the -i, -c, and -s options are only effective at invocation and no-ops
 when used with `set' -- if you've ever wondered how you get an 80-page man
 `page', that's how -- but would that really have made a difference?  It's
 only there for convenience, after all, and has no effect.


You can avoid these surprises by making -i just as other options, i.e.,
working within the shell not just when a shell is started. The document
should be complete. Why it matters you have a 8, 80, or 800 manpages? Few
people prints it. After all, most people just search it and it is better to
make it complete.

I believe the best way to make the manpage short is to make the
implementation intuitive so that people can derive the behavior in complex
cases based on what are described in simpler cases in the manual. With this
in mind, one avoids the description of complexes cases in the manual.

 -i is described in the OPTIONS section of the man page, since invocation
  is the only place using it makes sense.
 
  Not necessarily. See my other example sent recently on the mailing
  list of trying to get COLUMNS.

 With regards to COLUMNS, one doesn't need interactive mode to do that.
 There are lots of ways to set COLUMNS to the screen width.  It's much more
 productive to state your requirements clearly so people can suggest
 solutions -- people here are more than happy to help -- than to complain
 that the shell isn't satisfying your assumptions.

 Chet
 --
 ``The lyf so short, the craft so long to lerne.'' - Chaucer
  ``Ars longa, vita brevis'' - Hippocrates
 Chet Ramey, ITS, CWRUc...@case.edu javascript:;
 http://cnswww.cns.cwru.edu/~chet/



-- 
Regards,
Peng

Re: -i option of set missing in man bash

[Peng Yu]

 The -i option obviously works with set. But it is missing in the man
 page. Should this be added?

 No.  It's really only there for completeness, so things like `set $-'
 work as expected without error.

But if something is in the implementation, it should be also in the
documentation, right?

  The -s option does the same thing,
 for example.

 -i is described in the OPTIONS section of the man page, since invocation
 is the only place using it makes sense.

Not necessarily. See my other example sent recently on the mailing
list of trying to get COLUMNS.

One may want to manually set -i option in a bash script for whatever
reason. (In this case, it is to check COLUMNS.)

-- 
Regards,
Peng

Re: -i option of set missing in man bash

[Peng Yu]

On Thu, Mar 12, 2015 at 1:29 PM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Thu, Mar 12, 2015 at 01:13:18PM -0500, Peng Yu wrote:
 One may want to manually set -i option in a bash script for whatever
 reason. (In this case, it is to check COLUMNS.)

 http://mywiki.wooledge.org/BashFAQ/091

This does not work if one need to use

#!/usr/bin/env bash


-- 
Regards,
Peng

-i option of set missing in man bash

[Peng Yu]

Hi,

The -i option obviously works with set. But it is missing in the man
page. Should this be added?

~$ echo $-
himBH
~$ set +i
~$ echo $-
hmBH

The following lines are from the man page.

   set [--abefhkmnptuvxBCEHPT] [-o option-name] [arg ...]
   set [+abefhkmnptuvxBCEHPT] [+o option-name] [arg ...]

-- 
Regards,
Peng

Re: How to deal with errors in ()?

[Peng Yu]

On Mon, Mar 9, 2015 at 2:07 PM, Chet Ramey chet.ra...@case.edu wrote:
 On 3/8/15 6:05 PM, Stephane Chazelas wrote:

 Are bash questions no longer on topic here? bash-bug used to be
 the place to discuss bash (before help-bash was created). It maps to the
 gnu.bash.bug newsgroup. I don't think help-bash maps to usenet
 (though you can access it over NNTP on gmane).

 I personally don't care which mailing list gets the questions.

 $ bash --norc
 bash-4.3$ echo (exit 123)
 /dev/fd/63
 bash-4.3$ echo $!
 12142
 bash-4.3$ wait $!
 bash: wait: pid 12142 is not a child of this shell

 Having the process substitution pid in $! is not very useful if
 you can't wait for it to retrieve the status.

 I agree.  I will look at making that work for the next release of bash.

Would making an environment array (just as PIPE_STATUS) that will be
expanded to the process IDs of the all executed background commands
started from the last command be helpful to my original question? (But
this should also work with commands connected by pipes in order to
solve the problem.)

-- 
Regards,
Peng

Where is the usage for x in a b c; { echo $x; } documented in bash man page?

[Peng Yu]

Hi,

The following code works in bash.

for x in a b c; { echo $x; }

But I only find the following in bash man page. Does anybody know
where the above usage is documented? Thanks.

for name [ [ in [ word ... ] ] ; ] do list ; done

-- 
Regards,
Peng

The correct way to use for without polluting the environment

[Peng Yu]

Hi, I use unset to remove x from the environment once the for loop is
finished. Is it the best way to do in bash? Thanks.

for x in a b c
do
  echo $x
done
unset x

-- 
Regards,
Peng

How to deal with errors in ()?

[Peng Yu]

Hi,

http://mywiki.wooledge.org/ProcessSubstitution

The above webpage says the following.

commandA (commandB; [commandB's exit code is available here from $?])
[commandB's exit code cannot be obtained from here.  $? holds
commandA's exit code]

But I am wondering if there is a walkaround to deal with errors in
(). The ideal behavior should be that if there is a error in (),
then we should not consider commandA is executed correctly even if its
return status is 0.

Does anybody have a good solution for this situation? Thanks.

-- 
Regards,
Peng

echo does not follow the getopt convention

[Peng Yu]

Hi,

The following shows that echo does not following the getopt
convection. Is it better to make all bash internal command following
the getopt convention?

~$ echo -n
~$ echo -n -n
~$ echo -n -- -n # I think that the output should just be -n not -- -n.
-- -n~$

http://software.frodo.looijaard.name/getopt/

-- 
Regards,

Peng Yu, Ph.D.
Assistant Professor
Dept. Electrical and Computer Engineering  TEES-AgriLife Center for
Bioinformatics and Genomic Systems Engineering (CBGSE)
Texas AM University
Office: 215F WEB
Phone: (979) 320-9822
Email: pe...@tamu.edu
http://yubiolab.wordpress.com

cd won't change the prompt if the command is typed in the vi editor (vi mode).

[Peng Yu]

Hi,

I have vi mode set.

set -o vi.

Then I type v to enter the vi editor. In the vi editor, I type 'cd
/tmp'. Then, I get the following screen output. Notice that the prompt
does not change immediately after the cd command. Could anybody
reproduce this behavior? Thanks.

~$
cd /tmp
~$ echo $PWD
/tmp
/tmp$

-- 
Regards,
Peng

Re: Specify completion without name

[Peng Yu]

 Presumably you would also include aliases, shell builtins, and functions
 in this file.

Yes. I just want to replace executables in PATH by the result from my
custom function. I think that aliases, builtins, and functions are all
in the memory of bash already, so it doesn't take extra time to search
them from bash. Therefore, the ideal behaviors is that bash still
search for aliases, builtins and functions, and in addition check the
results returned by my custom function. Hence, my custom function
should not return anything about them (this should be expect at least
by default).

-- 
Regards,
Peng

Re: Specify completion without name

[Peng Yu]

 I would envision that such a completion function would assemble its list
 of possible completions by using your read-from-a-file mechanism and
 augment the list using compgen -a/compgen -b/compgen -A function.  It
 would probably also want to handle glob patterns and expand them to
 potentially multiple completions, but that gets tricky.

I did not know that it is so simple to get the alias (compgen -a),
buildins (compgen -b) and functions (compgen -A function) as you
mentioned. Once I know these, I agree with you that bash need not
handle these internally, rather user can call these three functions
directly. But beware to clearly document these by giving working
EXAMPLE code which include these three commands (not just text
explanation without working code, by working code I mean code
snippet is discouraged, a complete completion function should be
provided).

BTW, as I mentioned several times the bash man favors document
maintainer rather readers. For example, the following help doesn't
help me much when I want to learn how to use compgen.

~$ help compgen
compgen: compgen [-abcdefgjksuv] [-o option] [-A action] [-G globpat]
[-W wordlist] [-P prefix] [-S suffix] [-X filterpat] [-F function] [-C
command] [word]
Display the possible completions depending on the options.  Intended
to be used from within a shell function generating possible completions.
If the optional WORD argument is supplied, matches against WORD are
generated.

The manpage also use a reference rather than list all the options
directly. Readers have to jump to complete to understand how to use
compgen. This is also inconvenient to users.

  Generate possible completion matches for word according  to  the
  options,  which  may  be  any  option  accepted  by the complete
  builtin


If you consider it repetitive to discuss the same option twice in both
compgen and complete, at least, you can expand help compgen to
describe all the options (merge the current description of compgen and
complete in man). Other help messages are so concise that they are not
very helpful for learning how to use them. I'd suggest change all of
them as well.

-- 
Regards,
Peng

Re: Specify completion without name

[Peng Yu]

 The bash man page already has ~70 pages manual. I don't like it to grow to
 ~700 pages (like the ABS Guide) with all the working examples you expected.
 :)

Do you use search at all? :) If you use search, it doesn't really
matter if is a 700 page manual.

-- 
Regards,
Peng

Specify completion without name

[Peng Yu]

Hi,

I want to customize the command completion for completing executables,
I want to search in a file (which includes all the executables in
PATH) rather than the default PATH variable. But I don't see how to do
so, as the following help indicates that it can only configure how to
complete the arguments of a command. Does anybody know how to
configure command completion for the name itself rather than its
arguments? Thanks!

~/Downloads$ help complete
complete: complete [-abcdefgjksuv] [-pr] [-o option] [-A action] [-G
globpat] [-W wordlist] [-P prefix] [-S suffix] [-X filterpat] [-F
function] [-C command] [name ...]
For each NAME, specify how arguments are to be completed.
If the -p option is supplied, or if no options are supplied, existing
completion specifications are printed in a way that allows them to be
reused as input.  The -r option removes a completion specification for
each NAME, or, if no NAMEs are supplied, all completion specifications.

-- 
Regards,
Peng

Re: Specify completion without name

[Peng Yu]

 empty lines.  There is no programmable completion mechanism to complete
 on non-empty command names.

I'm wondering if it is worthwhile to add such a feature. I have run
into the problem that it is very slow to command complete an
incomplete command especially when other programs are accessing the
disk (note that I frequently open new terminals, so the caching done
within a bash process does not help me much).

If I can configure how to complete on non-empty command names, I could
just check a file with all the commands in PATH are stored. By
checking just a single file, presumable the search can be much faster
than searching many directories.

-- 
Regards,
Peng


-- 
Regards,
Peng

Is the description of set -- missing in man bash or at least difficult to find?

[Peng Yu]

Hi,

As I mentioned previously, there are shortcomings in man bash. Here, I
just point another example. And I hope my suggestion will be
addressed.

As a reasonable search strategy to search for how to set $@ is to
search for '$@' in man bash. The literal word '$@' appears at the
following locations.

...performed,  with the exception of $@ as explained below under Special...
...expands to a separate word.  That is, $@ is equivalent to $1...
...When there are no positional parameters, $@ and $@  expand  to...
...of $@ and ${name[@]} as explained above (see PARAMETERS)...

Search for 'set --' returns nothing. And search for '--' is prohibited
as there are too many of them. If the manual discusses how to 'set
$@', then it is at least hard to find. My suggestion is to add a new
paragraph to the existing discussion of $@, so that by searching $@,
readers can easily see how set $@.

   @  Expands  to  the positional parameters, starting from one.  When
  the  expansion  occurs  within  double  quotes,  each  parameter
  expands to a separate word.  That is, $@ is equivalent to $1
  $2 ...  If the double-quoted expansion occurs within  a  word,
  the  expansion  of the first parameter is joined with the begin-
  ning part of the original word, and the expansion  of  the  last
  parameter  is  joined  with  the last part of the original word.
  When there are no positional parameters, $@ and $@  expand  to
  nothing (i.e., they are removed).

As others pointed out LDP/abs is a more readable document, the
following link is a much better document on helping me to find how to
set $@.

http://tldp.org/LDP/abs/html/internalvariables.html

-- 
Regards,
Peng

Re: Is the description of set -- missing in man bash or at least difficult to find?

[Peng Yu]

 Second, just search for the 'set' builtin, near the bottom of the man page.

Thank for clarifying the usage of set.

I looked closely to the document of set. I just find another problem,
it says the following. However, the description of -- way down below.
It should be the option be described. A rule of thumb is that the
order of the description of the options should be the same as the
order in the following line.

 set [--abefhkmnptuvxBCEHPT] [-o option-name] [arg ...]

But my main point is if a person only knows the keyword $@, he will
not be able to find how to set it in the manual. The description of
'set --' should also be added to the description of $@ to explain how
to set $@.

Also, If no arguments follow this option, then the positional
parameters are unset.  Otherwise, the positional parameters are set to
the args, even if some of them begin with a -.
should be rewritten as
If no arguments follow this option, then the positional parameters $@
are unset.  Otherwise, the positional parameters $@ are set to the
args, even if some of them begin with a -.
so that $@ is searchable.

-- 
Regards,
Peng

Re: Is the description of set -- missing in man bash or at least difficult to find?

[Peng Yu]

 +1 vote on getting the parameters listed with a leading dollar sign.
 The individual single character is difficult to search for but the
 combination of $@ and so forth for the others is a useful search
 string.  I have often wanted the manual to include the $@
 combination instead of just the @ name.

I agree $@ is better than @.

But not to repeat myself again, did you get my major point that I have
restated to DJ Mills?

-- 
Regards,
Peng

Re: Is the description of set -- missing in man bash or at least difficult to find?

[Peng Yu]

 There are shortcomings in _the man documentation format_ and one of them
 is that it doesn't work (at least for me...) when the documentation is
 longer than one screen or thereabouts. I've pretty much come to the
 conclusion that any man page that is over a couple of hundred lines is
 a waste of my time and should probably not even exist in the first
 place.

Hi CJ,

You didn't get my point. Please see my reply to DJ Mills, if my first
email is not clear enough to you.

The shortcomings is not in the man format. Even if it is in the man
format, it can be compensated in the way that I mentioned.

-- 
Regards,
Peng

Re: [Help-bash] How to keep only files match the prefix when do command completion?

[Peng Yu]

Hi Greg,

 New users do not mess with programmable completion.

Given the context, I though that it was clear that new users means
users new to command completion. If it was not clear, I make it
explicit  here.

-- 
Regards,
Peng

Ill positioned 'until' keyword

[Peng Yu]

Hi,

I looks a little wired why 'until' is the way it is now. According to
the manual until is before the do-done block.

until test-commands; do consequent-commands; done

A common design of until in other language is that it allows the loop
body be executed at least once and test the condition at the end of
the run of the first time. It seems that a better of bash should also
follow the practice. If I understand it correctly, the above is exact
the same as the following, in which case the do done block can be
executed zero time. Because of this, I think that the current 'until'
is not necessary, and probably better to change its definition so that
it allows the execution of the loop at least once.

while ! test-commands; do consequent-commands; done


In short, I'd expect the following code echo 9 (not working with the
current bash).

COUNTER=9
do
 echo COUNTER $COUNTER
 let COUNTER-=1
done
until [  $COUNTER -lt 10 ];


I'd like to hear for what reason 'until' is designed in the way it is
now. Shall we considered to at least allow an option in bash to change
it meaning to the one I propose (or adding a different command, like
until2, for what I proposed), which give us time to let the orignal
until usage dies out.

-- 
Regards,
Peng

Re: How to protect and interpret it later on? (w/o using eval)

[Peng Yu]

 THAT will work.  But why are you writing a script to read a shell command
 and then execute it?  There is already a program that reads shell commands

This capability will be useful for debugging bash script.

For example, I have a set of commands in a bash script, each of them
output some thing to stdout. However, I can not tell the start and end
of each output. I could have a command that does what I expect
execute.sh to do.

cmd1
cmd2
...

I can easily append each command with 'execute.sh', which will not
only tell me the boundary of each output but also help me remember
what command was executed.

execute.sh cmd1
execute.sh cmd2
...

Since what I expect is not possible, the next solution is to use the
eval version which requires to put quote around cmd1, cmd2... But this
is very annoying, when cmd1, cmd2 themselves have the quotation marks,
I have to escape these marks.

So neither solution to execute.sh is ideal, although any of them is
better than the other in certain situations.

-- 
Regards,
Peng

Re: How to protect and interpret it later on? (w/o using eval)

[Peng Yu]

 WHAT are you trying to DO?

I think that you might completely miss my point. I try to explain it
better. Let me know if this time it makes more sense to you.

I want to execute any command as if the 'execute.sh' does not present,
except that I want to print the command so that I know want the
command is executed. (This can be used when I call several commands in
a script and I know what part of the output associated to what
command). E.g. I can run

ls  /tmp/tmp.txt

When I call,

execute.sh ls  /tmp/tmp.txt

I want it actually to do

echo ls  /tmp/tmp.txt
ls  /tmp/tmp.txt


Note that I could define execute.sh such that

execute.sh ls  /tmp/tmp.txt

means

echo ls  /tmp/tmp.txt
eval ls  /tmp/tmp.txt

But this interface of execute.sh is not as good the previous one.

Note that there could be other symbols that bash normal process, such as '21'.

I'm looking for a general solution, Pierre's answer is not as general as I want.

The FAQ http://mywiki.wooledge.org/BashFAQ/050 doesn't really answer
my question.

BTW, where is the help-bash mailing list mentioned (at least not on
bash home page)? I have never seen it before.

http://tiswww.case.edu/php/chet/bash/bashtop.html

-- 
Regards,
Peng

How to protect and interpret it later on? (w/o using eval)

[Peng Yu]

Hi,

~$ cat ../execute.sh
#!/usr/bin/env bash

echo $@
$@

$  ../execute.sh  ls /tmp/tmp.txt
$ cat /tmp/tmp.txt #I don't want ls be in the file
ls
main.sh

'' will not work unless eval is used in execute.sh.

$ ../execute.sh  ls '' /tmp/tmp.txt
ls  /tmp/tmp.txt
ls: cannot access : No such file or directory
/tmp/tmp.txt

How to make execute protect  and interpret it later on w/o using eval?

-- 
Regards,
Peng

How to directly modify $@?

[Peng Yu]

Hi,

I don't see if there is a way to directly modify $@. I know 'shift'.
But I'm wondering if there is any other way to modify $@.

~$ 1=x
-bash: 1=x: command not found
~$ @=(a b c)
-bash: syntax error near unexpected token `a'

-- 
Regards,
Peng

Re: converting array to string by quoting each element for eval

[Peng Yu]

Hi Greg,

 **NEVER** use getopt(1).  It is broken.  It cannot be made to work
 correctly.  Its entire design is flawed.

I don't see these warnings in my systems (macports and ubuntu) (This
is version of getopt on macports and ubuntu is free, I don't see there
is a reason that getopt can not be ported to the two systems that you
mentioned). All I see that is relevant is the following. I don't think
that just because that it has a BUGS section in the manpage, it can be
called broken. man bash also has BUGS section, is bash considered as
broken?

BUGS
   getopt(3) can parse long options with optional arguments that
are given an empty optional argument (but can not do this for short
options). This  getopt(1)  treats
   optional arguments that are empty as if they were not present.

   The syntax if you do not want any short option variables at all
is not very intuitive (you have to set them explicitely to the empty
string).

AUTHOR
   Frodo Looijaard fr...@frodo.looijaard.name


Note that it doesn't mean that I am resistant to getopts. It is just
that I don't think that your logic is valid. But I will read more
about getopts to see if it is necessary to convert from getopt to
getopts.

 ./myscript . -type f -name '*'

To support your claim, tell me what myscript would be if the commands

./myscript 'a b' 'c d' -type f -name '*
./myscript 'a b' -type f -name '*

actually do

find 'a b' 'a d' -maxdepth 1 -type f -name '*
find 'a b' -maxdepth 1 -type f -name '*

-- 
Regards,
Peng

Re: converting array to string by quoting each element for eval

[Peng Yu]

 And that is enough of this nonsense.  I have cited three official manuals
 for you already.  Let's move on.

I don't get it. Do you mean both traditional getopt and Debian getopt
are broken. To me it seems that Debian getopt is made to address the
short coming of transitional getopt. Yet you still think Debian getopt
is broken?

 Now that we know the goal, it's simple enough:

No. That is not my goal. It is just a simplification of my goal. It is
really hard to enumerate all the possible use cases, but I will try.

In additional to the use cases I stated in the previous email, let's
consider some more:

 ./myscript 'a b' 'c d' -o output.txt -type f -name '*'

or

 ./myscript 'a b' 'c d' -output output.txt -type f -name '*'

or

 ./myscript -output output.txt -type f -name '*' 'a b' 'c d'

or

 ./myscript -type f -name '*' 'a b' -output output.txt  'c d'

is equivalent

 find 'a b' 'a d' -maxdepth 1 -type f -name '*'  output.txt


-- 
Regards,
Peng

Re: converting array to string by quoting each element for eval

[Peng Yu]

 You may safely use getopts (the builtin).  Never getopt.

If my understanding is correct, 'getopts' doesn't support the long
format. Hence, it does not satisfy my need and I shall not use it.

-- 
Regards,
Peng

converting array to string by quoting each element for eval

[Peng Yu]

Hi,

I find that I have to make a program quotearg.sh to convert an array
to a string by quoting each element. So that it be used for eval.

I'm not sure if there is a way that I can do eval in bash without
having to use quotearg.sh. If there is no such a way, should
quotearg.sh be added in bash (if it is not available in bash yet), as
it provides a fundamental functionality?


~/linux/bin/src/bash/quotearg/main$ cat ./main.sh
#!/usr/bin/env bash

../quotearg.sh a b c
../quotearg.sh ' ' ' ''
../quotearg.sh 'a' 'a b'

echo 

args=('a' 'a b')
cmd=printf 'x%sx\n' ${args[@]}
eval $cmd

echo #the following is what I want, the above is not.

args=('a' 'a b')
arg_string=`../quotearg.sh ${args[@]}`
cmd=printf 'x%sx\n' $arg_string
eval $cmd

~/linux/bin/src/bash/quotearg/main$  ./main.sh
'a' 'b' 'c'
''\''' ' ' ''
'a' 'a b'

xax
xax
xbx

xax
xa bx



-- 
Regards,
Peng

Re: converting array to string by quoting each element for eval

[Peng Yu]

   Why not use the array instead of making it into a single string?

 $cmd ${args[@]}

   Why are you using eval or quotearg.sh? It sounds as if you are
   making the process more complicated than it need be.

For the examples that I gave, probably it is not necessary.

I'm yet to make a concrete complex example to demonstrate its
usefulness in practice. But I think that it is easy to imagine that
there will be some complex commands that are composed by concatenating
multiple strings to be evaled, in which case this may be useful.

-- 
Regards,
Peng

Re: converting array to string by quoting each element for eval

[Peng Yu]

    In any case, combining a command and its arguments in a single
    string is almost always the wrong way to go about it.

Please compare the two scripts and see if the second one makes more sense.

/tmp$ cat not_convert_args_to_string.sh
#!/bin/bash

options=$2
find $1 $options
echo find $1 $options
/tmp$ cat convert_args_to_string.sh
#!/bin/bash

options=$2
cmd=find $1 $options
eval $cmd
echo $cmd
/tmp$ ./not_convert_args_to_string.sh . -type f -name '*'
find: `./cvcd': Permission denied
find . -type f -name '*'
/tmp$ ./convert_args_to_string.sh . -type f -name '*'|head
find: `./cvcd': Permission denied
./.main.sh.swp
./0001e4ecf6175
./alm.log
./amt.log
./convert_args_to_string.sh
./FLEXnet/11.5.0.0 build 56285Macrovision
./FLEXnet/524288-16F7558F-328B-4dc3-BEDF-095C1F14FFF1
./FLEXnet/524288-34E9EE98-50ED-4c6c-BD0F-F539123FD064
./FLEXnet/524288-85A0F138-527D-4012-8175-79A3AEA4152E
./FLEXnet/freCqflgCxFrwiBvBiCadibCwg



-- 
Regards,
Peng

Re: converting array to string by quoting each element for eval

[Peng Yu]

On Tue, Nov 15, 2011 at 6:43 PM, Chris F.A. Johnson
ch...@cfajohnson.com wrote:
 On Tue, 15 Nov 2011, Peng Yu wrote:

    In any case, combining a command and its arguments in a single
    string is almost always the wrong way to go about it.

 Please compare the two scripts and see if the second one makes more sense.

 /tmp$ cat not_convert_args_to_string.sh
 #!/bin/bash

 options=$2
 find $1 $options
 echo find $1 $options

  More sensible would be to have each option a separate argument and
  do:

 location=$1
 shift
 find $location $@

No. My real example use getopt. If I have each option in a separate
argument, I need to know all the possible arguments to find, which is
not a viable route.

 /tmp$ cat convert_args_to_string.sh
 #!/bin/bash

 options=$2
 cmd=find $1 $options
 eval $cmd
 echo $cmd

   See above.

 /tmp$ ./not_convert_args_to_string.sh . -type f -name '*'
 find: `./cvcd': Permission denied
 find . -type f -name '*'

   Use 'set -x' to see exactly what your script is doing.

How to pass the option -type f -name '*' correctly?

/tmp$ ./not_convert_args_to_string.sh . -type f -name '*'
+ options='-type f -name '\''*'\'''
+ find . -type f -name ''\''*'\'''
find: `./cvcd': Permission denied
+ echo find . -type f -name ''\''*'\'''
find . -type f -name '*'

-- 
Regards,
Peng

Why complete doesn't print anything if it is called in a bash script?

[Peng Yu]

Hi,

It is strange to me why complete doesn't print anything when it is
called in a bash script. I must have misunderstood some fundamentals.
Does anybody know why? Thanks!

~$ cat main.sh
#!/usr/bin/env bash

. ~/.bashrc

complete

~$ ./main.sh
~$ complete |head
complete -F _kill kill
complete -F _renice renice
complete -F _smbpasswd smbpasswd
complete -F _postconf postconf
complete -F _ldapwhoami ldapwhoami
complete -F _ldapaddmodify ldapadd
complete -F _launchctl launchctl
complete -F _java java
complete -F _stream stream
complete -F _filedir_xspec oodraw


-- 
Regards,
Peng

Re: Why complete doesn't print anything if it is called in a bash script?

[Peng Yu]

On Sat, Nov 12, 2011 at 10:01 AM, Andreas Schwab sch...@linux-m68k.org wrote:
 Peng Yu pengyu...@gmail.com writes:

 It is strange to me why complete doesn't print anything when it is
 called in a bash script. I must have misunderstood some fundamentals.
 Does anybody know why? Thanks!

 If complete does not print anything then there are no completions
 defined.

The question is why. I have source my bashrc, as you can see there are
completions defined in the interactive shell.

-- 
Regards,
Peng

Re: Why complete doesn't print anything if it is called in a bash script?

[Peng Yu]

On Sat, Nov 12, 2011 at 10:18 AM, Chet Ramey chet.ra...@case.edu wrote:
 On 11/12/11 10:41 AM, Peng Yu wrote:
 Hi,

 It is strange to me why complete doesn't print anything when it is
 called in a bash script. I must have misunderstood some fundamentals.
 Does anybody know why? Thanks!

 Since complete happily shows completions when run from a shell script,
 there must be code in your bashrc that prevents them from being
 defined if the shell is not interactive.

Thank you for reminding me! That's indeed the case (shown below is
from the bashrc file).

case $- in
  *i*) [[ -f /opt/local/etc/bash_completion ]]  .
/opt/local/etc/bash_completion ;;
esac

-- 
Regards,
Peng

Customize the command resolution in bash?

[Peng Yu]

Hi,

bash by default searchs in paths specified in the environment variable
PATH (separated by :). I'm not aware if there is any cache mechanism
to save the run time (but even so, different terminals still can not
see the same cache, hence each terminal has the overhead to create the
cache). When there are many files in PATH, it is going to slow down
the performance.

One simple remedy is to instead search in a file where the abspaths of
all the commands are saved (of course, this database file can be
generated by using the command 'find' to search for all the
directories in $PATH, which process can be scheduled to run
periodically using cron). To make this work, I'm wondering if there is
an easy way to customize the way that bash resolve a command.

-- 
Regards,
Peng

invoke tilde expansion on quoted string

[Peng Yu]

Hi,

I know from the document that tilde expansion only works if the string
is unquoted (see below)

~$ cd '~/..'
-bash: cd: ~/..: No such file or directory
~$ cd ~/..
/Users$

I'm wondering if I already have a string variable, is there a bash
native to do tilde expansion on it.

var='~/..'
cd $var#how to change this line?

-- 
Regards,
Peng

Re: What is the correct way to set up login environment in crontab?

[Peng Yu]

On Wed, Nov 9, 2011 at 7:45 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Tue, Nov 08, 2011 at 09:46:37PM -0600, Peng Yu wrote:
 I need to use cron to run some job. I know that cron only set up very
 basic environment. I'd like to duplicate my login environment.

 Just source /etc/profile and your ~/.bash_profile or ~/.profile (or
 whatever) from the script that your cron job executes.

 Personally I would advise against this.  Login environments are for
 interactive logins, not cron jobs.

So neither -i nor -l is necessary?

-- 
Regards,
Peng

Re: What is the correct way to set up login environment in crontab?

[Peng Yu]

On Wed, Nov 9, 2011 at 7:45 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Tue, Nov 08, 2011 at 09:46:37PM -0600, Peng Yu wrote:
 I need to use cron to run some job. I know that cron only set up very
 basic environment. I'd like to duplicate my login environment.

 Just source /etc/profile and your ~/.bash_profile or ~/.profile (or
 whatever) from the script that your cron job executes.

 Personally I would advise against this.  Login environments are for
 interactive logins, not cron jobs.

I sourced my ~/.bashrc, which source some other files. It seems the
environment variables defined in these files are not seen with env.
Why is so?


-- 
Regards,
Peng

Re: What is the correct way to set up login environment in crontab?

[Peng Yu]

On Wed, Nov 9, 2011 at 10:41 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Wed, Nov 09, 2011 at 10:29:52AM -0600, Peng Yu wrote:
 I sourced my ~/.bashrc, which source some other files. It seems the
 environment variables defined in these files are not seen with env.
 Why is so?

 Without seeing the code?  Impossible to say.  But you're doing it backwards.
 ~/.bashrc should be sourced FROM ~/.bash_profile.  A login shell reads
 ~/.bash_profile only, so it's the responsibility of ~/.bash_profile to
 read ~/.bashrc to set up aliases, functions, shopts, and other ephemeral
 shell settings that can't be inherited from the environment.

Sorry for the confusion. ~/.bash_profile is not the problem here. I
have the following line in /path/programtorun.sh

. ~/.bashrc

In ~/.bashrc, I have . ~/bash_some. In ~/.bash_some, I have some
variable assignment VAR=blah.

However, VAR is not seen in env in /path/programtorun.sh (called from cron).

-- 
Regards,
Peng

What is the correct way to set up login environment in crontab?

[Peng Yu]

Hi,

I need to use cron to run some job. I know that cron only set up very
basic environment. I'd like to duplicate my login environment. But
some environment variables are still not seen when I set the following
crontab entry. Does anybody know how to correctly set up the login
enviroment? (The manual describes what files are sourced, but I don't
want to manually source these files. I'm looking for a simpler way.)

* * * * * /bin/bash -i -l -c '/path/programtorun.sh  /tmp/my.log'


-- 
Regards,
Peng

Re: What is the best to pass an array with specially characters as command line arguments?

[Peng Yu]

Hi Clark,

 What do you mean by 1 long argument?

 [bash-4.2.10] # cat foo.sh
 v=  a b c ( a'b | 
 set -o noglob
 a=( $v )
 set +o noglob
 for i in ${a[@]}; do
     echo $i
 done
 [bash-4.2.10] # bash foo.sh
 a
 b
 c
 (
 a'b
 |
 [bash-4.2.10] #


I misunderstood the usage of ${args[@]}. I though it returns only
one long argument   a b c ( a'b | , but it actually expanded to 6
short arguments a, b, c, (, a'b and  |. Thanks for
clarification.



-- 
Regards,
Peng

Re: What is the best to pass an array with specially characters as command line arguments?

[Peng Yu]

On Mon, Nov 7, 2011 at 8:29 AM, Dennis Williamson
dennistwilliam...@gmail.com wrote:
 On Mon, Nov 7, 2011 at 7:23 AM, Peng Yu pengyu...@gmail.com wrote:
 Hi Clark,

 What do you mean by 1 long argument?

 [bash-4.2.10] # cat foo.sh
 v=  a b c ( a'b | 
 set -o noglob
 a=( $v )
 set +o noglob
 for i in ${a[@]}; do
     echo $i
 done
 [bash-4.2.10] # bash foo.sh
 a
 b
 c
 (
 a'b
 |
 [bash-4.2.10] #


 I misunderstood the usage of ${args[@]}. I though it returns only
 one long argument   a b c ( a'b | , but it actually expanded to 6
 short arguments a, b, c, (, a'b and  |. Thanks for
 clarification.



 --
 Regards,
 Peng



 If you use ${args[*]} (with quotes and an asterisk instead of an at
 sign), the result is one long argument. Otherwise, it's split.

Thanks! I didn't notice this difference before.


-- 
Regards,
Peng

Re: What is the best to pass an array with specially characters as command line arguments?

[Peng Yu]

Hi Clark,

  v=  a b c ( a'b | 
  a=( $v )
  echo ${a[@]}

 There's a @ char here.

I see. It's my mistake.

But I want to pass the 6 short arguments instead of 1 long argument to
echo. (echo is just an example, it can be any command that accepts
multiple arguments.)


~$ cat ./main1.sh
#!/usr/bin/env bash

#set -o noglob
verbatim_string=  a b c ( a'b | 

args=( $verbatim_string )
#set +o noglob

echo ${args[@]}

~$  ./main1.sh
a b c ( a'b |


-- 
Regards,
Peng

bash-completion between do and done

[Peng Yu]

Hi,

Current, bash doesn't do command completion between do and done (for loop).
I'm wondering if this feature can be added.

-- 
Regards,
Peng

How to automatically load alias from .bashrc in a bash script?

[Peng Yu]

Hi,

I want to use some alias in a bash script. But I have to include the
following line in the script in order to use the alias defined in
~/.bashrc. Is there a way to automatically load the alias from .bashrc
so that I don't have to explicitly include these two lines?

shopt -s expand_aliases
. ~/.bashrc


-- 
Regards,
Peng

Re: How to match regex in bash? (any character)

[Peng Yu]

On Thu, Sep 29, 2011 at 7:22 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Wed, Sep 28, 2011 at 12:43:01PM -0800, Roger wrote:
 Seems I used 'man regex' as well here.  AKA regex(3).  But I did
 realize this a few weeks ago; the real regex description being 'man 7 regex'.
 The Bash Manual Page denotes only regex(3).

 You're relatively fortunate that it's *that* easy to find on Linux.  On
 Linux, regex(3) points directly to regex(7), and you're done.

 On HP-UX, regex(3X) points to regcomp(3C) which points to regexp(5) which
 contains the actual definitions.

 On OpenBSD, regex(3) doesn't even *have* a SEE ALSO section; it's a dead
 end.  And regcomp(3) is the same page as regex(3), so that doesn't help
 either.  One would have to backtrack entirely, perhaps to grep(1).
 However, buried deep in the regex(3) page is a reference to re_format(7)
 (not even boldface).  And re_format(7) has the definitions, but getting
 there takes perseverance.  (For the record, grep(1) does point straight
 to re_format(7).)

 So you see, bash(1) *cannot* just link directly to regex(7), because
 that's not actually the correct final destination on most operating
 systems.  It's only correct on Linux.  Bash uses the regex(3) library
 interface, so that is the correct place for bash to refer the reader.

Therefore, either bash manpage should specify clearly which regex
manpage it should be in each system (which a bad choice, because there
can be a large number of systems), or the bash manpage should omit all
the non consistent reference and say something like see more details
in info or something else that is platform independent. Referring to
regex(3) without any quantification is not a very good choice .

-- 
Regards,
Peng

Re: How to match regex in bash? (any character)

[Peng Yu]

On Thu, Sep 29, 2011 at 10:38 AM, Chet Ramey chet.ra...@case.edu wrote:
 On 9/29/11 9:48 AM, Peng Yu wrote:

 Therefore, either bash manpage should specify clearly which regex
 manpage it should be in each system (which a bad choice, because there
 can be a large number of systems), or the bash manpage should omit all
 the non consistent reference and say something like see more details
 in info or something else that is platform independent. Referring to
 regex(3) without any quantification is not a very good choice .

 Why, exactly?  regex(3) is the one thing that's portable across systems,
 it happens to describe the interfaces bash uses, and it contains the
 appropriate system-specific references.  `info' is considerably less
 portable and widespread than `man', so a man page reference is the best
 choice.

We all have discovered that regex(3) is not consistent across all the
platform. Why you say it is portable?

As I mentioned previously, the best is to add a few examples in man
bash. Based on the assumption that you don't want to add an example in
man bash, then the next choice to add a reference to info, even though
it may not always be available in all the system (but as least it
should be downloadable from bash gnu website).

-- 
Regards,
Peng

Re: How to match regex in bash? (any character)

[Peng Yu]

On Thu, Sep 29, 2011 at 11:06 AM, Greg Wooledge wool...@eeg.ccf.org wrote:
 On Thu, Sep 29, 2011 at 10:59:19AM -0500, Peng Yu wrote:
 We all have discovered that regex(3) is not consistent across all the
 platform. Why you say it is portable?

 The three systems I mentioned earlier today all have regex(3).  Which
 system have you found, which doesn't have it?

I think that I misunderstood some of the previous emails.

However, on ubuntu, there is regex(3) and regex(7). Based on the
context in man bash, regex(7) is more relevant than regex(3), although
regex(3) does mention extend regular expression, it is more of a
document for the C interface.

When it is used, the string to the right of the operator is
considered an extended regular expression and matched accordingly (as
in regex(3))

Also, regex(3) does not mention the difference between $x =~ .txt
and $x=~ .txt. I think that the difference should be addressed
in man bash.

Bottom line, regex(3) is not a good manpage to refer in the above
sentence. It is better to think of other alternative rather than
trying to justify we should stuck with it.

 As I mentioned previously, the best is to add a few examples in man
 bash.

 I would not object to that, but I can't speak for Chet.

 Another option would be to refer to the POSIX definition of
 Extended Regular Expressions as a web site.  I wish they had
 better URLs, though.  The URL I have for it at the moment is
 http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap09.html#tag_09_04




-- 
Regards,
Peng