Re: Missing separator error can be caused by too many files in alternate recipe format
It turns out it is not a bug--just general error/quirk. Okay, I found the error, and the reason why I been going insane over this over the past few hours. My filenames have semi-colons in them. That is what triggered the "missing separator" error. Makefile.am: # filename with semi-colons template_file = data;sectionfoo;foo-1 dist_pkgdata_DATA = $(template_file) # generated Makefile rule here would cause make crash $(dist_pkgdata_DATA): Makefile @echo 'no-op for datafile' $@ My solution is to use the transformation program option of Autotools (via configure.ac). Sorry for the inconvenience. -- penguindude24
Re: Missing separator error can be caused by too many files in alternate recipe format
My filenames have semi-colons in them and yes I did escape them (as-is text): admin_files = some-file.dat admin_templates_files = Admin/templates/adminLoginForm;admin;default Admin/templates/blockEdit\;admin\;default Admin/templates/colorEdit\;admin\;default Admin/templates/data\;admin\;default Admin/templates/editbuttons\;admin\;default Admin/templates/editFilter\;admin\;default Admin/templates/editStory\;admin\;default Admin/templates/ispellok\;admin\;default Admin/templates/keywordEdit\;admin\;default Admin/templates/listFilters\;admin\;default Admin/templates/listStories\;admin\;default Admin/templates/messages\;admin\;default Admin/templates/otherLinks\;admin\;default Admin/templates/relatedlinks\;admin\;default Admin/templates/siteInfo\;admin\;default Admin/templates/templateEdit\;admin\;default Admin/templates/titles\;admin\;default Admin/templates/topicEdit\;admin\;default Admin/templates/varEdit\;admin\;default dist_admin_DATA = $(admin_files) # make dies here dist_adminTemplates_DATA = $(admin_templates_files) CORRECTION # causes problems dist_adminTemplates_DATA = $(admin_templates_files) $(dist_adminTemplates_DATA): Makefile @echo 'no-op for datafile' $@
Re: Missing separator error can be caused by too many files in alternate recipe format
On 05/27/2014 05:56 PM, PenguinDude24 wrote: On 05/27/2014 07:21 AM, Paul Smith wrote: On Mon, 2014-05-26 at 20:46 -0400, PenguinDude24 wrote: What I think happened is that obviously there was some internal error with heuristics that GNU Make (and most likely others), and the heuristics engine could not figure out how to parse that long line (maybe was not expecting data to be so long, or programmer so stupid to do that) and threw the error. GNU make supports an arbitrary number of targets and prerequisites, up to the size of your computer's memory. There are no (known) hardcoded limits. And, an error like "missing separator" is not what I would expect to see if the problem were related to some out of bounds error. Instead, I expect that either there's some issue with the way automake generated the makefile, or else some issue with the filenames in the target list, or a bug in GNU make related to heap management. The only way to know is if you give us the details necessary to understand the problem. What version of automake are you using? What version of GNU make? And, what is the (exact) contents of the line which is causing make to throw an error (and a few lines before/after it, for context)? It could be the way Automake generated the Makefile. Although I had an issue with this, I resolved it by omitting a recipe to make the data file (both Make and Automake). No data files are "built," but just needed them to be moved into pkgdatadir by Autotools. I cannot put raw sources in here because of confidentiality reasons, but here is a "foo example" of what I have with real raw text changed. Makefile.am: # -- pluginsdir = $(pkgdatadir)/plugins dist_plugins_DATA = plugin-1 plugin-2 messagesPlugindir = $(pluginsdir)/messages messagesPluginTemplatesdir = $(messagesPlugindir)/templates # Here is the supposed text that causes issues. This project I'm on has hundreds of template files (not my idea) messages_actual_data_files = data-file-1 data-file-2 ... data-file-300 .. data-file-nnn dist_messagesPluginTemplates_DATA = $(messages_actual_data_files) # FORM1 -- recipe that caused GNU Make crash, i.e "missing separator line nnn' # The target here is over two hundred or so data files (templates). The # generated line is pretty printed by automake it seems, because it has # formatting I did not put in there. $(dist_messagesPluginTemplates_DATA): Makefile @echo 'no-op for data file' $@ # FORM2 -- no crash with this one, but rule is not run because Makefile # exists and is newer than dependencies. Makefile: $(dist_messagesPluginTemplates_DATA) @echo 'no-op for data file' $@ I must add that in the generated Makefile, the output seems to be pretty printed like so: # line 345_example $(dist_directory_DATA) = \ target_foo-1 \ target_foo-2 \ target_foo-3 \ ... \ target_foo-n And Make complains of error on line 345_example. Also, in the Makefile.am that I made the variable declaration for that target, I added some minor pretty printing, and automake error checking (nor GNU Emacs)did not complain about it, and it ran until that line. Then the make(1) crash. Hit me up if you have any Q's. Just did some invistigation. Does not appear to be length related, seems to be data filename related. My filenames have semi-colons in them and yes I did escape them (as-is text): admin_files = some-file.dat admin_templates_files = Admin/templates/adminLoginForm;admin;default Admin/templates/blockEdit\;admin\;default Admin/templates/colorEdit\;admin\;default Admin/templates/data\;admin\;default Admin/templates/editbuttons\;admin\;default Admin/templates/editFilter\;admin\;default Admin/templates/editStory\;admin\;default Admin/templates/ispellok\;admin\;default Admin/templates/keywordEdit\;admin\;default Admin/templates/listFilters\;admin\;default Admin/templates/listStories\;admin\;default Admin/templates/messages\;admin\;default Admin/templates/otherLinks\;admin\;default Admin/templates/relatedlinks\;admin\;default Admin/templates/siteInfo\;admin\;default Admin/templates/templateEdit\;admin\;default Admin/templates/titles\;admin\;default Admin/templates/topicEdit\;admin\;default Admin/templates/varEdit\;admin\;default dist_admin_DATA = $(admin_files) # make dies here dist_adminTemplates_DATA = $(admin_templates_files)
Re: Missing separator error can be caused by too many files in alternate recipe format
On 05/27/2014 07:21 AM, Paul Smith wrote: On Mon, 2014-05-26 at 20:46 -0400, PenguinDude24 wrote: What I think happened is that obviously there was some internal error with heuristics that GNU Make (and most likely others), and the heuristics engine could not figure out how to parse that long line (maybe was not expecting data to be so long, or programmer so stupid to do that) and threw the error. GNU make supports an arbitrary number of targets and prerequisites, up to the size of your computer's memory. There are no (known) hardcoded limits. And, an error like "missing separator" is not what I would expect to see if the problem were related to some out of bounds error. Instead, I expect that either there's some issue with the way automake generated the makefile, or else some issue with the filenames in the target list, or a bug in GNU make related to heap management. The only way to know is if you give us the details necessary to understand the problem. What version of automake are you using? What version of GNU make? And, what is the (exact) contents of the line which is causing make to throw an error (and a few lines before/after it, for context)? It could be the way Automake generated the Makefile. Although I had an issue with this, I resolved it by omitting a recipe to make the data file (both Make and Automake). No data files are "built," but just needed them to be moved into pkgdatadir by Autotools. I cannot put raw sources in here because of confidentiality reasons, but here is a "foo example" of what I have with real raw text changed. Makefile.am: # -- pluginsdir = $(pkgdatadir)/plugins dist_plugins_DATA = plugin-1 plugin-2 messagesPlugindir = $(pluginsdir)/messages messagesPluginTemplatesdir = $(messagesPlugindir)/templates # Here is the supposed text that causes issues. This project I'm on has hundreds of template files (not my idea) messages_actual_data_files = data-file-1 data-file-2 ... data-file-300 .. data-file-nnn dist_messagesPluginTemplates_DATA = $(messages_actual_data_files) # FORM1 -- recipe that caused GNU Make crash, i.e "missing separator line nnn' # The target here is over two hundred or so data files (templates). The # generated line is pretty printed by automake it seems, because it has # formatting I did not put in there. $(dist_messagesPluginTemplates_DATA): Makefile @echo 'no-op for data file' $@ # FORM2 -- no crash with this one, but rule is not run because Makefile # exists and is newer than dependencies. Makefile: $(dist_messagesPluginTemplates_DATA) @echo 'no-op for data file' $@ I must add that in the generated Makefile, the output seems to be pretty printed like so: # line 345_example $(dist_directory_DATA) = \ target_foo-1 \ target_foo-2 \ target_foo-3 \ ... \ target_foo-n And Make complains of error on line 345_example. Also, in the Makefile.am that I made the variable declaration for that target, I added some minor pretty printing, and automake error checking (nor GNU Emacs)did not complain about it, and it ran until that line. Then the make(1) crash. Hit me up if you have any Q's.
Missing separator error can be caused by too many files in alternate recipe format
The typical format for recipes is: Target: Dependencies @echo 'we are cooking your favorite app' But when you do this: Dependencies: Target @echo 'we are cooking your favorite app' you run the risk of getting the "missing separator error on line nnn" if the "Dependencies" list is too long--I think my app has a few hundred data files or so. I am working on this project, and that error kept me down for a few hours. What I think happened is that obviously there was some internal error with heuristics that GNU Make (and most likely others), and the heuristics engine could not figure out how to parse that long line (maybe was not expecting data to be so long, or programmer so stupid to do that) and threw the error. Anyway, it's a good case to look at! I also think this should be added to the Automake documentation, but I don't know Texinfo yet :)
Re: Long file lists and deep directioriies with Automake
I also had to use the install-data-hook target to get the data files nested just right also.
Re: Long file lists and deep directioriies with Automake
On 12/17/2013 04:05 PM, Gavin Smith wrote: On Mon, Dec 2, 2013 at 8:21 PM, PenguinDude24 wrote: Automake generates the Makefile.in files, and configure makes the 'Makefile' for the directory. But the generated makefiles give errors. GNU Make gives the ominous 'missing separator error'. That errors can be triggered for a variety of reasons. Long file list for targets and deep nested dirs are at least two of the reasons. Are you sure those are the reasons, and there aren't extra newlines being inserted in these long lines? Can you post the relevant lines of the Makefiles? Yea, I'm sure or by my own analysis. I'm also using GNU Emacs, and Emacs is anal when it is in Automake or Makefile modes. Certain Emacs modes have error checking and it did not warn me -- likely no simple error detected. Emacs has saved my bacon many times by alerting me to simple errors that may have taken up too much time. Many times GNU Emacs told me "suspicious line 44 in file. Want to save anyway?" I look at the line and there is some syntax highlighting to tell me what is up with the file. I did some internet searches and came across some "programmer sites" (you know all of the popular ones). Some guys were having issues with the 'missing separator' error message. The software project I was working on required many data files to be installed -- hundreds of them. I mean a few hundred files (most of them only a few bytes, but they had structured file names for clarity). And since Automake does not support wildcards for adding targets or data files to lists (says that in the documentation), you have to specify file names manually, so that Automake can make rules for them accordingly. By adding the files manually, I found out that files that were down to the third level, i.e "dir-1/dir-2/dir-3/foo-datafile.bin", I got 'missing separator' error. I used the 'nobase_dist_pkgdata_DATA' directive to schedule Automake to write out the data to Makefiles. When the list to that directive contained too many data files, I also got the 'missing file separator error' from GNU Make. The Makefile.am looks like this: # Makefile.am -- Automake file for sub-dir foo/data/normalized # Over a hundred files added this way. Not hand typed -- just copy and paste from terminal :) nobase_dist_pkgdata_DATA = foo-datfile-0.bin foo-datfile-1.bin foo-datefile-n.bin foo foo-datefile-N.bin # Of course, blank rule so that generated makefiles will not complain about 'no rule to make foo target' $(nobase_dist-pkgdata_DATA): @echo 'Doing NO-OP for datafile ' $@ Generated Makefile croaked. My solution was to break up the long file lists into smaller ones, and used 'SUBDIR' more often. Meh, dispite the limitation, I still used Autotools. I'm on Debian by the way.
Re: Automake Digest, Vol 133, Issue 5
On 12/16/2013 12:00 PM, automake-requ...@gnu.org wrote:
Send Automake mailing list submissions to
automake@gnu.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.gnu.org/mailman/listinfo/automake
or, via email, send a message with subject or body 'help' to
automake-requ...@gnu.org
You can reach the person managing the list at
automake-ow...@gnu.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Automake digest..."
Today's Topics:
1. Re: automake breaks my file by putting includes after rules;
how do I fix this? (Jason Gross)
--
Message: 1
Date: Mon, 16 Dec 2013 10:49:49 -0500
From: Jason Gross
To: Gavin Smith
Cc: Automake Mailing List
Subject: Re: automake breaks my file by putting includes after rules;
how do Ifix this?
Message-ID:
Content-Type: text/plain; charset=ISO-8859-1
These compiled files get distributed/installed, so I think they belong in
DATA.
The solution I ended up using was manually including an "all-am" target
below the definition, which overrides the one that automake generates.
I think the other contributors to the project would complain only slightly
less about depending on automake than they did about depending on
autoreconf (I ended up setting up an extra branch of the source repo which
the autogenerated files get automatically pushed to); the intended
contributors/users of the source repo are mathematicians who may have
little to no programming experience.
-Jason
On Mon, Dec 2, 2013 at 8:10 AM, Gavin Smithwrote:
On Wed, Nov 20, 2013 at 9:33 PM, Jason Gross
wrote:
I have a Makefile.am which includes another file in order to get a list
of
files it should compile (I don't want to require running autoreconf and
configure whenever you add a file that the makefile already knows how to
compile). But this does not work with automake's generated all and
all-am
targets, which compute their dependencies on the basis of the DATA
variable, which is auto-defined to be nobase_hott_DATA, which depends on
variables set in the included file. How do I fix this? (Should I
submit a
bug report that includes should come before the autogenerated targets?)
Is it possible to use the "all-local" target to compile these files,
and use another variable to list the files instead of DATA?
--
___
First off, forcing your own 'all-am' target is considered gross
malpractice. Your going to hurt yourself with that. Do not do that. The
recommended course for your case is to use a target, namely
'install-data-hook' in the source Makefile.am. BUT, that is in most
cases the worst case scenario. Based on what you told me, I think that
you can get by with the regular Autotools facilites. Why else would you
use Autotools?
Read along to find out the regular and recommended way.
You did not provide any scrubbed example of what you are working with,
so I will rub something together.
# filename -- Makefile.am
# -
# ... lines in the Makefile.am before this stage, such as
# SUBDIR directive, or other directive like LTLIBRARIES or
# other AutoMake directives (that will generate stuff in
# the generated Makefiles).
# We list the data files for your package -- just an example.
# Note, the dist prefix addition on the DATA directive is mandatory
# else your data files will not be distributed. DATA files are not
# shipped by defualt!
dist_pkgdata_DATA = root-datafile-1.bin correlations.csv \
data-f0.bin data-f1.bin data-f2.bin
# Then, you must include a 'dummy rule' (even though its a 'dummy rule')
# to make that list of files. Here is the rule to build
# dist_pkgdata_DATA items.
# --
$(dist_pkgdata_DATA):
@echo 'Dummy rule NO-OP for: ' $@
Without the dummy rule, the generated Makefile(s) are worthless. GNU
Make (and all the rest of them) will tell you, or something similar, "No
rule to make 'targetname' " and exit with fatal error.
For the grand finale, when you run './configure && make && make install'
cmds, the data files will reside in ( prefix = '/usr/local' and
datarootdir = "$prefix/$share"):
In bash run:
ls
$(prefix)/$(datarootdir)/$(your_package_name_in_configure.ac)/{root-datafile-1.bin,correlations.csv,
... etc}
Then, it is up to your application to find the data files in
/usr/local/share/your-pkg-name/datafile-1.bin, and the rest of them.
And when you are using autoreconf, you have to export certain variables
into the environment for them to work. There are many env vars that it
can take, but the ones that are vital are:
AUTOCONF
AUTOHEADER
AUTOM4TE
AUTOMAKE
ACLOCAL
LIBTOOLIZE
Else, when you run autoreconf, the versions it runs may be different
because there are many auto*foo* versions installed in one system at one
time. T
Re: automake breaks my file by putting includes after rules; how do I fix this?
On 12/16/2013 12:00 PM, automake-requ...@gnu.org wrote:
Send Automake mailing list submissions to
automake@gnu.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.gnu.org/mailman/listinfo/automake
or, via email, send a message with subject or body 'help' to
automake-requ...@gnu.org
You can reach the person managing the list at
automake-ow...@gnu.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Automake digest..."
Today's Topics:
1. Re: automake breaks my file by putting includes after rules;
how do I fix this? (Jason Gross)
--
Message: 1
Date: Mon, 16 Dec 2013 10:49:49 -0500
From: Jason Gross
To: Gavin Smith
Cc: Automake Mailing List
Subject: Re: automake breaks my file by putting includes after rules;
how do Ifix this?
Message-ID:
Content-Type: text/plain; charset=ISO-8859-1
These compiled files get distributed/installed, so I think they belong in
DATA.
The solution I ended up using was manually including an "all-am" target
below the definition, which overrides the one that automake generates.
I think the other contributors to the project would complain only slightly
less about depending on automake than they did about depending on
autoreconf (I ended up setting up an extra branch of the source repo which
the autogenerated files get automatically pushed to); the intended
contributors/users of the source repo are mathematicians who may have
little to no programming experience.
-Jason
On Mon, Dec 2, 2013 at 8:10 AM, Gavin Smithwrote:
On Wed, Nov 20, 2013 at 9:33 PM, Jason Gross
wrote:
I have a Makefile.am which includes another file in order to get a list
of
files it should compile (I don't want to require running autoreconf and
configure whenever you add a file that the makefile already knows how to
compile). But this does not work with automake's generated all and
all-am
targets, which compute their dependencies on the basis of the DATA
variable, which is auto-defined to be nobase_hott_DATA, which depends on
variables set in the included file. How do I fix this? (Should I
submit a
bug report that includes should come before the autogenerated targets?)
Is it possible to use the "all-local" target to compile these files,
and use another variable to list the files instead of DATA?
--
___
First off, forcing your own 'all-am' target is considered gross
malpractice. Your going to hurt yourself with that. Do not do that. The
recommended course for your case is to use a target, namely
'install-data-hook' in the source Makefile.am. BUT, that is in most
cases the worst case scenario. Based on what you told me, I think that
you can get by with the regular Autotools facilites. Why else would you
use Autotools?
Read along to find out the regular and recommended way.
You did not provide any scrubbed example of what you are working with,
so I will rub something together.
# filename -- Makefile.am
# -
# ... lines in the Makefile.am before this stage, such as
# SUBDIR directive, or other directive like LTLIBRARIES or
# other AutoMake directives (that will generate stuff in
# the generated Makefiles).
# We list the data files for your package -- just an example.
# Note, the dist prefix addition on the DATA directive is mandatory
# else your data files will not be distributed. DATA files are not
# shipped by defualt!
dist_pkgdata_DATA = root-datafile-1.bin correlations.csv \
data-f0.bin data-f1.bin data-f2.bin
# Then, you must include a 'dummy rule' (even though its a 'dummy rule')
# to make that list of files. Here is the rule to build
# dist_pkgdata_DATA items.
# --
$(dist_pkgdata_DATA):
@echo 'Dummy rule NO-OP for: ' $@
Without the dummy rule, the generated Makefile(s) are worthless. GNU
Make (and all the rest of them) will tell you, or something similar, "No
rule to make 'targetname' " and exit with fatal error.
For the grand finale, when you run './configure && make && make install'
cmds, the data files will reside in ( prefix = '/usr/local' and
datarootdir = "$prefix/$share"):
In bash run:
ls
$(prefix)/$(datarootdir)/$(your_package_name_in_configure.ac)/{root-datafile-1.bin,correlations.csv,
.. etc}
Then, it is up to your application to find the data files in
/usr/local/share/your-pkg-name/datafile-1.bin, and the rest of them.
And when you are using autoreconf, you have to export certain variables
into the environment for them to work. There are many env vars that it
can take, but the ones that are vital are:
AUTOCONF
AUTOHEADER
AUTOM4TE
AUTOMAKE
ACLOCAL
LIBTOOLIZE
Else, when you run autoreconf, the versions it runs may be different
because there are many auto*foo* versions installed in one system at one
time. Th
Long file lists and deep directioriies with Automake
I'm working on a project. Actually, I'm rennovating a project. And its mostly script files that do not need to be compiled with a C or C++ compiler; were talking Perl here. Anyway, the package requries that many data files be installed. I'm telling Automake via Makefile.am to put them into the computed 'pkgdatadir'. And the app requires that files within a deep directory tree be installed. The whole app is like that. (This is a very popular app form 2001 that you all have heard of.) For example: # --- app_template_files_list-1 = somedir1/somedir2/somdir3/actual_file.dat somedir1/somedir2/somdir3/someothe_file.dat [some more files here like those ...] Automake generates the Makefile.in files, and configure makes the 'Makefile' for the directory. But the generated makefiles give errors. GNU Make gives the ominous 'missing separator error'. That errors can be triggered for a variety of reasons. Long file list for targets and deep nested dirs are at least two of the reasons. And if GNU Make gives errors. I'm sure other Makes out there will make your computer explode. So I'm assuming all makes will give errors. My work around for this is to used an install-data-hook in the Makefile.am. I used the configure computed values for pkgdatadir, and use the computed values for $(INSTALL) and $(MKDIR_P). So my install-data-hook rule looks like this: install-data-hook: $(MKDIR_P) $(pkgdatadir)/appdir1/appdir2/appdir3 $(INSTALL) $(somedatafile_list0) $(pkgdatadir)/appdir1/appdir2/appdir3 $(INSTALL) $(somedatafile_list1) $(pkgdatadir)/appdir1/appdir2/appdir3 $(INSTALL) $(somedatafile_list2) $(pkgdatadir)/appdir1/appdir2/appdir3 Is this good Autoconf/Automake practice?
