This is a respin of the patches that I sent out in September under the title of "pre-lexer-2". It adds UTF-8 support, at John's suggestion.
Ben Pfaff (5): str: Inline some trivial functions. str: Change "char" to "byte" in function names. str: Add some functions for handling UTF-8. identifier: Add some functions for Unicode syntax. command: Factor command name matching out of command.c. Smake | 4 + src/data/data-in.c | 76 +++--- src/data/dictionary.c | 2 +- src/data/format-guesser.c | 18 +- src/data/identifier.c | 82 ++++++- src/data/identifier.h | 9 +- src/data/settings.c | 10 +- src/data/sys-file-writer.c | 16 +- src/language/command.c | 385 +++++------------------------- src/language/control/repeat.c | 8 +- src/language/data-io/data-list.c | 4 +- src/language/data-io/data-parser.c | 14 +- src/language/data-io/data-reader.c | 6 +- src/language/data-io/get-data.c | 4 +- src/language/data-io/print.c | 6 +- src/language/expressions/operations.def | 2 +- src/language/expressions/parse.c | 4 +- src/language/lexer/automake.mk | 5 +- src/language/lexer/command-name.c | 235 ++++++++++++++++++ src/language/lexer/command-name.h | 46 ++++ src/language/lexer/format-parser.c | 10 +- src/language/lexer/lexer.c | 18 +- src/language/syntax-string-source.c | 2 +- src/language/utilities/host.c | 2 +- src/language/utilities/set.q | 14 +- src/language/utilities/title.c | 4 +- src/libpspp/argv-parser.c | 8 +- src/libpspp/message.c | 2 +- src/libpspp/model-checker.c | 4 +- src/libpspp/str.c | 306 ++++++++++++++---------- src/libpspp/str.h | 59 ++++- src/output/ascii.c | 4 +- src/ui/gui/text-data-import-dialog.c | 36 ++-- src/ui/syntax-gen.c | 20 +- tests/automake.mk | 17 ++- tests/language/command.at | 2 +- tests/language/data-io/inpt-pgm.at | 4 +- tests/language/lexer/command-name-test.c | 149 ++++++++++++ tests/language/lexer/command-name.at | 234 ++++++++++++++++++ 39 files changed, 1200 insertions(+), 631 deletions(-) create mode 100644 src/language/lexer/command-name.c create mode 100644 src/language/lexer/command-name.h create mode 100644 tests/language/lexer/command-name-test.c create mode 100644 tests/language/lexer/command-name.at _______________________________________________ pspp-dev mailing list [email protected] http://lists.gnu.org/mailman/listinfo/pspp-dev
