https://github.com/gregvw created https://github.com/llvm/llvm-project/pull/168420
None >From 1b7812ee09b3bef929f44acfe10f35e4fabb7531 Mon Sep 17 00:00:00 2001 From: Claude <[email protected]> Date: Mon, 17 Nov 2025 17:57:54 +0000 Subject: [PATCH 1/2] Add design documents for customizable functions feature This commit adds comprehensive design and test plan documents for implementing customizable functions in Clang using the 'custom' keyword. The feature will allow developers to write: custom void do_thing(auto& t) { t.method(); } Instead of manually writing CPO boilerplate with tag_invoke pattern. Documents include: - CustomizableFunctionsDesign.md: Complete design specification with implementation plan, semantics, syntax, and open questions - CustomizableFunctionsTestPlan.md: Detailed testing strategy covering parser, semantic, codegen, integration, and performance tests The implementation follows similar patterns to coroutines and other transformative language features in Clang. --- clang/docs/CustomizableFunctionsDesign.md | 1037 +++++++++++++++++++ clang/docs/CustomizableFunctionsTestPlan.md | 744 +++++++++++++ 2 files changed, 1781 insertions(+) create mode 100644 clang/docs/CustomizableFunctionsDesign.md create mode 100644 clang/docs/CustomizableFunctionsTestPlan.md diff --git a/clang/docs/CustomizableFunctionsDesign.md b/clang/docs/CustomizableFunctionsDesign.md new file mode 100644 index 0000000000000..e11d7c794c819 --- /dev/null +++ b/clang/docs/CustomizableFunctionsDesign.md @@ -0,0 +1,1037 @@ +# Customizable Functions Design Document + +**Author:** TBD +**Date:** 2025-11-17 +**Status:** Draft + +## Table of Contents + +1. [Introduction](#introduction) +2. [Motivation](#motivation) +3. [Proposed Syntax](#proposed-syntax) +4. [Semantics](#semantics) +5. [Implementation Plan](#implementation-plan) +6. [Testing Strategy](#testing-strategy) +7. [Open Questions](#open-questions) +8. [References](#references) + +--- + +## Introduction + +This document proposes adding a new language feature to Clang: **Customizable Functions** using the `custom` keyword. This feature would allow library authors to easily create Customization Point Objects (CPOs) using the tag_invoke pattern without manually writing extensive boilerplate code. + +### Goals + +- Reduce boilerplate for implementing the tag_invoke CPO pattern +- Improve library interface design and customization points +- Maintain compatibility with existing C++20/23 code +- Generate efficient, zero-overhead abstractions + +### Non-Goals + +- Replacing existing customization mechanisms (ADL, specialization) +- Changing the behavior of tag_invoke itself +- Adding runtime dispatch overhead + +--- + +## Motivation + +### Current State: Manual CPO Implementation + +Library authors currently need to write significant boilerplate to create proper CPOs: + +```cpp +namespace my_lib { + namespace CPO_DETAIL { + // Default implementation + void do_thing(auto& t) { t.method(); } + + // Functor class + struct do_thing_fn { + // tag_invoke overload + template <typename T> + constexpr auto operator()(T& t) const + requires requires { tag_invoke(*this, t); } + { + return tag_invoke(*this, t); + } + + // Fallback overload + template <typename T> + constexpr auto operator()(T& t) const + requires (!requires { tag_invoke(*this, t); }) + { + return do_thing(t); + } + }; + } + + // Inline constexpr instance + inline constexpr CPO_DETAIL::do_thing_fn do_thing{}; +} +``` + +This pattern: +- Is verbose and error-prone +- Requires understanding of advanced C++ techniques +- Must be repeated for each customization point +- Is difficult to maintain and modify + +### Proposed State: Declarative Syntax + +With customizable functions, the same functionality becomes: + +```cpp +namespace my_lib { + custom void do_thing(auto& t) { + t.method(); + } +} +``` + +The compiler automatically generates the CPO boilerplate, making the code: +- More concise and readable +- Less error-prone +- Easier to maintain +- Self-documenting + +### Real-World Use Cases + +1. **Generic Algorithm Libraries**: Customization points for ranges, algorithms +2. **Serialization/Deserialization**: Custom serializers for user types +3. **Logging/Debugging**: Customizable formatting and output +4. **Resource Management**: Custom allocators, deleters +5. **Async/Await Patterns**: Customizable awaitable operations + +--- + +## Proposed Syntax + +### Basic Syntax + +```cpp +custom <return-type> <function-name>(<parameters>) <body> +``` + +### Examples + +#### Simple Free Function +```cpp +namespace lib { + custom void print(auto const& value) { + std::cout << value; + } +} +``` + +#### With Explicit Return Type +```cpp +custom int compute(int x, int y) { + return x + y; +} +``` + +#### Multiple Parameters +```cpp +custom void transform(auto& container, auto&& func) { + for (auto& elem : container) { + func(elem); + } +} +``` + +#### With Constraints +```cpp +custom void process(auto& t) + requires std::copyable<decltype(t)> +{ + t.process(); +} +``` + +#### Template Function +```cpp +template <typename T> +custom void serialize(T const& value, std::ostream& out) { + out << value; +} +``` + +### Syntax Restrictions + +- `custom` keyword must appear before the return type +- Cannot be used with: + - Member functions (initially; may be relaxed later) + - Constructors/destructors + - Operators (except when implementing operator() for the generated functor) + - Virtual functions +- Must have a function body (not just a declaration) +- Can be used in any namespace (including global) + +--- + +## Semantics + +### Transformation Overview + +When the compiler encounters a `custom` function, it performs the following transformation: + +1. **Create Detail Namespace** (optional, configurable) + - Named `CPO_DETAIL` or `<function_name>_detail` + - Contains the default implementation and functor class + +2. **Generate Default Implementation** + - Original function body becomes a hidden implementation function + - Used as the fallback when tag_invoke is not available + +3. **Generate Functor Class** + - Named `<function_name>_fn` + - Implements two operator() overloads: + - Primary: calls tag_invoke (when available) + - Fallback: calls default implementation + +4. **Create CPO Instance** + - Inline constexpr variable with original function name + - Type is the functor class + - This becomes the actual customization point + +### Generated Code Structure + +For a `custom` function: +```cpp +custom RetType func_name(Params...) { body } +``` + +The compiler generates: +```cpp +namespace CPO_DETAIL { + // Default implementation (hidden) + RetType func_name_impl(Params...) { body } + + // Functor class + struct func_name_fn { + // Primary overload: use tag_invoke if available + template <typename... Args> + constexpr auto operator()(Args&&... args) const + noexcept(noexcept(tag_invoke(*this, std::forward<Args>(args)...))) + requires requires { tag_invoke(*this, std::forward<Args>(args)...); } + { + return tag_invoke(*this, std::forward<Args>(args)...); + } + + // Fallback overload: use default implementation + template <typename... Args> + constexpr auto operator()(Args&&... args) const + noexcept(noexcept(func_name_impl(std::forward<Args>(args)...))) + requires (!requires { tag_invoke(*this, std::forward<Args>(args)...); } + && requires { func_name_impl(std::forward<Args>(args)...); }) + { + return func_name_impl(std::forward<Args>(args)...); + } + }; +} + +// The actual CPO +inline constexpr CPO_DETAIL::func_name_fn func_name{}; +``` + +### Name Lookup and ADL + +The generated CPO follows standard C++ name lookup rules: + +1. **Unqualified calls** to the CPO trigger ADL +2. **tag_invoke** is found via ADL in the namespace of the arguments +3. Users customize by defining `tag_invoke` in their namespace: + +```cpp +namespace user { + struct MyType { }; + + // Customization + void tag_invoke(lib::do_thing_fn, MyType& t) { + // Custom implementation + } +} + +// Usage +user::MyType obj; +lib::do_thing(obj); // Finds user::tag_invoke via ADL +``` + +### Overload Resolution + +The two operator() overloads in the functor are constrained to be mutually exclusive: +- Primary: `requires tag_invoke(*this, args...)` +- Fallback: `requires !tag_invoke(*this, args...) && default_impl(args...)` + +This ensures: +- No ambiguity during overload resolution +- tag_invoke always preferred when available +- Fallback only selected when tag_invoke is not viable + +### Template Instantiation + +For templated custom functions: +```cpp +template <typename T> +custom void process(T& value) { ... } +``` + +The functor class itself is templated on the CPO's template parameters, and the operator() remains templated on the call-site arguments. + +--- + +## Implementation Plan + +### Phase 1: Core Infrastructure (Minimal Viable Product) + +**Goal**: Get basic `custom` keyword parsing and simple code generation working. + +#### 1.1 Add Keyword and Language Option + +**Files**: +- `clang/include/clang/Basic/TokenKinds.def` +- `clang/include/clang/Basic/LangOptions.def` + +**Tasks**: +- [ ] Add `custom` as a C++20 keyword with flag `KEYCUSTOMFN` +- [ ] Add language option `CustomizableFunctions` (default: disabled) +- [ ] Add `-fcustomizable-functions` / `-fno-customizable-functions` flags + +**Testing**: +- [ ] Verify keyword is recognized when feature is enabled +- [ ] Verify keyword is not recognized when feature is disabled + +#### 1.2 Extend Parser + +**Files**: +- `clang/include/clang/Parse/Parser.h` +- `clang/lib/Parse/ParseDecl.cpp` +- `clang/include/clang/Sema/DeclSpec.h` + +**Tasks**: +- [ ] Add `isCustomFunction` flag to `DeclSpec` +- [ ] Modify `ParseDeclarationSpecifiers` to recognize `custom` keyword +- [ ] Set the flag when `custom` is encountered +- [ ] Ensure proper error handling for invalid uses + +**Testing**: +- [ ] Parse simple `custom void foo() {}` +- [ ] Reject `custom` on member functions +- [ ] Reject `custom` on declarations without definitions +- [ ] Proper error messages for invalid syntax + +#### 1.3 Extend AST + +**Files**: +- `clang/include/clang/AST/Decl.h` +- `clang/lib/AST/Decl.cpp` +- `clang/lib/AST/DeclPrinter.cpp` +- `clang/lib/AST/ASTDumper.cpp` + +**Tasks**: +- [ ] Add `IsCustomFunction` bit to `FunctionDeclBitfields` +- [ ] Add `isCustomFunction()` / `setCustomFunction()` methods +- [ ] Update `DeclPrinter` to print `custom` keyword +- [ ] Update `ASTDumper` to show custom function flag + +**Testing**: +- [ ] AST dump shows custom function annotation +- [ ] AST printer reproduces `custom` keyword + +#### 1.4 Basic Semantic Analysis + +**Files**: +- `clang/lib/Sema/SemaDecl.cpp` +- `clang/include/clang/Sema/Sema.h` + +**Tasks**: +- [ ] Create `ActOnCustomFunctionDecl()` hook +- [ ] Validate custom function constraints: + - Must have a body + - Cannot be member function + - Cannot be virtual + - Cannot be main() +- [ ] Mark function as custom in AST + +**Testing**: +- [ ] Semantic errors for invalid custom functions +- [ ] Accept valid custom function declarations + +### Phase 2: Code Generation (Core Transformation) + +**Goal**: Generate the CPO boilerplate during Sema. + +#### 2.1 Create Sema Transformation Module + +**Files** (new): +- `clang/include/clang/Sema/SemaCustomFunction.h` +- `clang/lib/Sema/SemaCustomFunction.cpp` + +**Tasks**: +- [ ] Implement `class CustomFunctionTransformer` +- [ ] Add transformation entry point: `TransformCustomFunction(FunctionDecl*)` +- [ ] Generate detail namespace (optional, via flag) +- [ ] Generate default implementation function +- [ ] Generate functor class with operator() overloads +- [ ] Generate inline constexpr CPO instance +- [ ] Integrate into `ActOnFunctionDeclarator` + +**Testing**: +- [ ] Generated AST contains all components +- [ ] Names are correct (e.g., `func_name_fn`) +- [ ] Proper linkage and storage for generated entities + +#### 2.2 Functor Class Generation + +**Tasks**: +- [ ] Create `CXXRecordDecl` for functor class +- [ ] Add `operator()` with tag_invoke call +- [ ] Add `operator()` with fallback call +- [ ] Generate proper constraints (requires clauses) +- [ ] Handle noexcept specifications +- [ ] Handle return type deduction + +**Testing**: +- [ ] Functor class has correct structure +- [ ] Overload resolution works correctly +- [ ] Constraints are mutually exclusive + +#### 2.3 CPO Instance Generation + +**Tasks**: +- [ ] Create `VarDecl` for CPO instance +- [ ] Set proper storage class (inline constexpr) +- [ ] Initialize with functor class instance +- [ ] Handle name shadowing of original function + +**Testing**: +- [ ] CPO instance has correct type +- [ ] CPO instance is constexpr +- [ ] Original function name resolves to CPO instance + +### Phase 3: Template Support + +**Goal**: Handle template custom functions correctly. + +#### 3.1 Template Function Transformation + +**Tasks**: +- [ ] Handle `template <...> custom void foo(...)` +- [ ] Preserve template parameters on generated entities +- [ ] Handle template instantiation correctly +- [ ] Support template constraints + +**Testing**: +- [ ] Template custom functions instantiate correctly +- [ ] Template argument deduction works +- [ ] SFINAE works with custom functions +- [ ] Constraints propagate correctly + +#### 3.2 Template Parameter Forwarding + +**Tasks**: +- [ ] Forward template parameters to functor class +- [ ] Ensure proper argument forwarding in operator() +- [ ] Handle perfect forwarding scenarios + +**Testing**: +- [ ] Forwarding references work correctly +- [ ] Move semantics preserved +- [ ] No extra copies/moves + +### Phase 4: Advanced Features + +#### 4.1 Customization Options + +Add attributes or keywords to control generation: + +```cpp +[[custom::no_detail_namespace]] +custom void foo() { } + +[[custom::detail_namespace("my_detail")]] +custom void bar() { } +``` + +**Tasks**: +- [ ] Parse customization attributes +- [ ] Support custom detail namespace names +- [ ] Support option to skip detail namespace +- [ ] Support custom functor class naming + +#### 4.2 Diagnostics and Error Messages + +**Tasks**: +- [ ] Improve error messages for invalid customizations +- [ ] Add notes pointing to original custom function +- [ ] Handle recursive tag_invoke calls +- [ ] Detect and warn about common mistakes + +**Testing**: +- [ ] Error messages are clear and helpful +- [ ] Source locations point to correct code +- [ ] Notes provide useful context + +#### 4.3 Integration with TInCuP Library + +**Tasks**: +- [ ] Option to generate TInCuP-compatible code +- [ ] Add `tincup::cpo_base` inheritance (optional) +- [ ] Generate concept traits (e.g., `foo_invocable_c`) +- [ ] Generate type traits (e.g., `foo_return_t`) + +### Phase 5: Tooling and Serialization + +#### 5.1 Modules and PCH Support + +**Files**: +- `clang/include/clang/Serialization/ASTWriter.h` +- `clang/include/clang/Serialization/ASTReader.h` + +**Tasks**: +- [ ] Serialize custom function flag +- [ ] Serialize generated entities +- [ ] Handle cross-module references + +**Testing**: +- [ ] Custom functions work in modules +- [ ] PCH files handle custom functions +- [ ] Module imports work correctly + +#### 5.2 Code Completion and IDE Support + +**Tasks**: +- [ ] Code completion for custom functions +- [ ] Signature help shows generated CPO +- [ ] Go-to-definition works for both original and generated code + +#### 5.3 Debug Info Generation + +**Tasks**: +- [ ] Generate proper debug info for all entities +- [ ] Debugger can step into both tag_invoke and fallback +- [ ] Variable inspection works correctly + +--- + +## Testing Strategy + +### Unit Tests + +#### Lexer/Parser Tests +**Location**: `clang/test/Parser/cxx-customizable-functions.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Basic parsing +custom void foo() { } // OK + +// Reject invalid uses +custom int x = 5; // expected-error {{expected function}} +class C { + custom void bar(); // expected-error {{custom functions cannot be member functions}} +}; +``` + +#### Semantic Analysis Tests +**Location**: `clang/test/SemaCXX/customizable-functions-sema.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +namespace test1 { + custom void func(int x) { } // OK +} + +namespace test2 { + custom void forward_decl(); // expected-error {{custom functions must have a body}} +} + +namespace test3 { + struct S { + custom void member(); // expected-error {{custom functions cannot be member functions}} + }; +} +``` + +#### AST Tests +**Location**: `clang/test/AST/customizable-functions-ast.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -ast-dump %s | FileCheck %s + +custom void test() { } + +// CHECK: FunctionDecl {{.*}} test 'void ()' +// CHECK-NEXT: CustomFunction +// CHECK: NamespaceDecl {{.*}} CPO_DETAIL +// CHECK: CXXRecordDecl {{.*}} test_fn +// CHECK: VarDecl {{.*}} test +``` + +### Integration Tests + +#### Code Generation Tests +**Location**: `clang/test/CodeGenCXX/customizable-functions-codegen.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -emit-llvm %s -o - | FileCheck %s + +custom void simple() { } + +// CHECK: @_ZN10CPO_DETAIL9simple_fnE = +// CHECK: define {{.*}} @_ZN10CPO_DETAIL11simple_implEv() +``` + +#### Tag Invoke Integration Tests +**Location**: `clang/test/SemaCXX/customizable-functions-tag-invoke.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +namespace lib { + custom void process(auto& x) { + x.default_process(); + } +} + +namespace user { + struct MyType { + void default_process() { } + void custom_process() { } + }; + + // Custom implementation + void tag_invoke(lib::process_fn, MyType& m) { + m.custom_process(); + } +} + +void test() { + user::MyType obj; + lib::process(obj); // Should call user::tag_invoke +} + +// Should compile without errors +``` + +#### Template Tests +**Location**: `clang/test/SemaCXX/customizable-functions-templates.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +template <typename T> +custom void serialize(T const& value) { + // default implementation +} + +namespace user { + struct MyType { }; + + template <typename T> + void tag_invoke(serialize_fn, MyType const& m) { + // custom implementation + } +} + +void test() { + user::MyType obj; + serialize(obj); // Should work +} +``` + +### End-to-End Tests + +#### Complete Example Test +**Location**: `clang/test/SemaCXX/customizable-functions-e2e.cpp` + +Full example with: +- Multiple custom functions +- User customizations via tag_invoke +- Overload resolution +- Template instantiation +- Constraint checking + +### Performance Tests + +#### Compilation Performance +- Measure compile time impact of custom functions +- Compare with manual CPO implementation +- Benchmark template instantiation overhead + +#### Runtime Performance +- Verify zero-overhead abstraction +- Compare generated code with manual implementation +- Benchmark with optimization levels -O0, -O2, -O3 + +### Regression Tests + +Add tests for: +- Previously reported bugs +- Edge cases discovered during development +- Interaction with other language features + +--- + +## Open Questions + +### 1. Detail Namespace Strategy + +**Question**: Should we always generate a detail namespace, or make it optional? + +**Options**: +- **A**: Always generate `CPO_DETAIL` namespace (consistent, but verbose) +- **B**: Optional via attribute: `[[custom::detail_namespace]]` +- **C**: Generate in same namespace without detail (simpler, but pollutes namespace) + +**Recommendation**: Option B - optional with default enabled + +### 2. Functor Class Naming + +**Question**: What naming convention for generated functor class? + +**Options**: +- **A**: `<name>_fn` (matches TInCuP) +- **B**: `<name>_functor` +- **C**: `__<name>_cpo_impl` (clearly internal) + +**Recommendation**: Option A - matches existing conventions + +### 3. Template Parameter Handling + +**Question**: How to handle template custom functions? + +**Options**: +- **A**: Functor class is templated, operator() is also templated +- **B**: Functor class is not templated, operator() is fully templated +- **C**: Generate a function template instead of functor class + +**Recommendation**: Option A - most flexible + +### 4. Member Function Support + +**Question**: Should we support custom member functions? + +**Options**: +- **A**: Never support (simplest) +- **B**: Support in future version (phase 2+) +- **C**: Support from the start + +**Recommendation**: Option B - defer to future + +**Rationale**: Member CPOs have different semantics and use cases. Start with free functions, add members later if needed. + +### 5. Interaction with Concepts + +**Question**: How should `requires` clauses on custom functions behave? + +```cpp +custom void foo(auto x) requires std::integral<decltype(x)> { } +``` + +**Options**: +- **A**: Constraint applies to both tag_invoke and fallback +- **B**: Constraint only applies to fallback +- **C**: Generate separate constraints for each + +**Recommendation**: Option A - constraint is part of the interface + +### 6. Noexcept Specifications + +**Question**: How to handle noexcept on custom functions? + +```cpp +custom void foo() noexcept { } +``` + +**Options**: +- **A**: Conditional noexcept in generated operator() +- **B**: Unconditional noexcept in operator() +- **C**: Ignore and always deduce + +**Recommendation**: Option A - preserve user intent + +### 7. Default Arguments + +**Question**: Should custom functions support default arguments? + +```cpp +custom void foo(int x = 0) { } +``` + +**Options**: +- **A**: Support (forward to operator()) +- **B**: Reject (error) +- **C**: Support only in fallback + +**Recommendation**: Option A - forward to both paths + +### 8. Customization Attributes + +**Question**: What attributes should we support for customization? + +**Proposals**: +- `[[custom::no_detail_namespace]]` +- `[[custom::detail_namespace("name")]]` +- `[[custom::functor_name("name")]]` +- `[[custom::tincup_compatible]]` +- `[[custom::inline_implementation]]` + +**Recommendation**: Start with namespace control, add others as needed + +--- + +## References + +### Academic Papers and Proposals + +1. **P1895R0**: tag_invoke: A general pattern for supporting customizable functions + https://wg21.link/p1895r0 + +2. **P2279R0**: We need a language mechanism for customization points + https://wg21.link/p2279r0 + +3. **P0443R14**: A Unified Executors Proposal for C++ + (Uses tag_invoke extensively) + +### Related Projects + +1. **TInCuP**: Tag Invoke Customization Points Library + https://github.com/sandialabs/TInCuP + +2. **niebloids and customization points**: + https://brevzin.github.io/c++/2020/12/19/cpo-niebloid/ + +### LLVM/Clang Resources + +1. **Coroutines Implementation**: `clang/lib/Sema/SemaCoroutine.cpp` +2. **Modules Implementation**: `clang/lib/Sema/SemaModule.cpp` +3. **Concepts Implementation**: `clang/lib/Sema/SemaConcept.cpp` + +### Similar Features in Other Languages + +1. **Rust Traits**: Customizable behavior via trait implementations +2. **Haskell Type Classes**: Similar customization mechanism +3. **Swift Extensions**: Protocol-based customization + +--- + +## Appendix A: Example Transformations + +### Example 1: Simple Function + +**Input**: +```cpp +namespace lib { + custom void log(auto const& msg) { + std::cout << msg << std::endl; + } +} +``` + +**Generated**: +```cpp +namespace lib { + namespace CPO_DETAIL { + void log_impl(auto const& msg) { + std::cout << msg << std::endl; + } + + struct log_fn { + template <typename T> + constexpr auto operator()(T const& msg) const + noexcept(noexcept(tag_invoke(*this, msg))) + requires requires { tag_invoke(*this, msg); } + { + return tag_invoke(*this, msg); + } + + template <typename T> + constexpr auto operator()(T const& msg) const + noexcept(noexcept(log_impl(msg))) + requires (!requires { tag_invoke(*this, msg); }) + { + return log_impl(msg); + } + }; + } + + inline constexpr CPO_DETAIL::log_fn log{}; +} +``` + +### Example 2: Template Function + +**Input**: +```cpp +template <typename Stream> +custom void write(Stream& stream, auto const& data) { + stream << data; +} +``` + +**Generated**: +```cpp +template <typename Stream> +struct write_fn { + template <typename S, typename T> + constexpr auto operator()(S& stream, T const& data) const + requires requires { tag_invoke(*this, stream, data); } + { + return tag_invoke(*this, stream, data); + } + + template <typename S, typename T> + constexpr auto operator()(S& stream, T const& data) const + requires (!requires { tag_invoke(*this, stream, data); }) + { + return write_impl(stream, data); + } +}; + +template <typename Stream> +inline constexpr write_fn<Stream> write{}; + +// Implementation +template <typename Stream> +void write_impl(Stream& stream, auto const& data) { + stream << data; +} +``` + +### Example 3: With Constraints + +**Input**: +```cpp +custom void process(auto& container) + requires requires { container.begin(); container.end(); } +{ + for (auto& elem : container) { + elem.update(); + } +} +``` + +**Generated** (constraints preserved in both paths): +```cpp +namespace CPO_DETAIL { + void process_impl(auto& container) + requires requires { container.begin(); container.end(); } + { + for (auto& elem : container) { + elem.update(); + } + } + + struct process_fn { + template <typename T> + constexpr auto operator()(T& container) const + requires requires { container.begin(); container.end(); } + && requires { tag_invoke(*this, container); } + { + return tag_invoke(*this, container); + } + + template <typename T> + constexpr auto operator()(T& container) const + requires requires { container.begin(); container.end(); } + && (!requires { tag_invoke(*this, container); }) + { + return process_impl(container); + } + }; +} + +inline constexpr CPO_DETAIL::process_fn process{}; +``` + +--- + +## Appendix B: Timeline and Milestones + +### Milestone 1: Proof of Concept (2-3 weeks) +- [ ] Keyword parsing +- [ ] Basic AST representation +- [ ] Simple transformation (single free function) +- [ ] Minimal test suite + +**Success Criteria**: Can compile and run a simple custom function example + +### Milestone 2: Core Functionality (4-6 weeks) +- [ ] Complete transformation logic +- [ ] Template support +- [ ] Comprehensive test suite +- [ ] Basic diagnostics + +**Success Criteria**: Can handle most common use cases correctly + +### Milestone 3: Production Ready (8-12 weeks) +- [ ] Advanced features +- [ ] Optimized codegen +- [ ] Full diagnostic support +- [ ] Documentation +- [ ] Integration tests + +**Success Criteria**: Ready for experimental use in real codebases + +### Milestone 4: Stable Release (12-16 weeks) +- [ ] Modules/PCH support +- [ ] Tooling integration +- [ ] Performance optimization +- [ ] Community feedback integration + +**Success Criteria**: Ready for upstream submission to LLVM + +--- + +## Appendix C: Potential Issues and Mitigations + +### Issue 1: Name Shadowing + +**Problem**: The CPO instance shadows the original function name. + +**Mitigation**: This is intentional - users call the CPO, not the original function. + +### Issue 2: Overload Set Conflicts + +**Problem**: Multiple custom functions with same name but different signatures. + +**Mitigation**: Generate a single functor class with multiple operator() overloads, similar to std::ranges algorithms. + +### Issue 3: Template Instantiation Bloat + +**Problem**: Each custom template function generates multiple templates. + +**Mitigation**: +- Inline small functions +- Use `if constexpr` where possible +- Rely on linker to deduplicate + +### Issue 4: Error Message Quality + +**Problem**: Errors in generated code may confuse users. + +**Mitigation**: +- Always point diagnostics to original source +- Add "note: in generated code for custom function 'X'" +- Suppress internal implementation details in errors + +### Issue 5: Compilation Time + +**Problem**: More templates may slow compilation. + +**Mitigation**: +- Measure and profile +- Optimize common patterns +- Consider precompiled headers for heavy users + +--- + +## Revision History + +- **2025-11-17**: Initial draft diff --git a/clang/docs/CustomizableFunctionsTestPlan.md b/clang/docs/CustomizableFunctionsTestPlan.md new file mode 100644 index 0000000000000..4fcc41bf44848 --- /dev/null +++ b/clang/docs/CustomizableFunctionsTestPlan.md @@ -0,0 +1,744 @@ +# Customizable Functions Test Plan + +This document provides a detailed testing strategy for the Customizable Functions feature. + +## Test Categories + +### 1. Parser Tests + +#### 1.1 Keyword Recognition +**File**: `clang/test/Parser/cxx-customizable-functions-keyword.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Test basic keyword recognition +custom void test1() { } // OK + +// Test without flag enabled +// RUN: %clang_cc1 -std=c++20 -fno-customizable-functions -fsyntax-only -verify=disabled %s +// disabled-error@-4 {{unknown type name 'custom'}} + +// Test with older C++ standard +// RUN: %clang_cc1 -std=c++17 -fcustomizable-functions -fsyntax-only -verify=cpp17 %s +// cpp17-warning@-8 {{'custom' is a C++20 extension}} +``` + +#### 1.2 Syntax Validation +**File**: `clang/test/Parser/cxx-customizable-functions-syntax.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Valid uses +custom void f1() { } +custom int f2() { return 0; } +custom auto f3() { return 42; } +custom void f4(int x) { } +custom void f5(auto x) { } + +namespace ns { + custom void f6() { } +} + +// Invalid: no body +custom void f7(); // expected-error {{custom functions must have a definition}} + +// Invalid: custom on non-function +custom int x = 5; // expected-error {{expected function}} +custom; // expected-error {{expected declaration}} + +// Invalid: multiple storage classes +static custom void f8() { } // expected-error {{'custom' cannot be combined with 'static'}} +extern custom void f9() { } // expected-error {{'custom' cannot be combined with 'extern'}} +``` + +#### 1.3 Declaration Specifier Ordering +**File**: `clang/test/Parser/cxx-customizable-functions-declspec-order.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Valid: custom before return type +custom void f1() { } +custom int f2() { return 0; } +custom constexpr int f3() { return 0; } + +// Valid: inline with custom +inline custom void f4() { } +custom inline void f5() { } + +// Invalid: custom after return type +void custom f6() { } // expected-error {{expected identifier}} +int custom f7() { return 0; } // expected-error {{expected identifier}} +``` + +### 2. Semantic Analysis Tests + +#### 2.1 Context Restrictions +**File**: `clang/test/SemaCXX/customizable-functions-context.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// OK: namespace scope +namespace ns { + custom void f1() { } +} + +// OK: global scope +custom void f2() { } + +// ERROR: class member +struct S1 { + custom void f3() { } // expected-error {{custom functions cannot be member functions}} + custom static void f4() { } // expected-error {{custom functions cannot be member functions}} +}; + +// ERROR: local scope +void test() { + custom void f5() { } // expected-error {{custom functions must be declared at namespace scope}} +} + +// ERROR: special members +struct S2 { + custom S2() { } // expected-error {{constructors cannot be custom}} + custom ~S2() { } // expected-error {{destructors cannot be custom}} + custom operator int() { return 0; } // expected-error {{conversion functions cannot be custom}} +}; + +// ERROR: virtual +struct S3 { + custom virtual void f6() { } // expected-error {{custom functions cannot be virtual}} +}; + +// ERROR: main function +custom int main() { return 0; } // expected-error {{'main' cannot be a custom function}} +``` + +#### 2.2 Template Validation +**File**: `clang/test/SemaCXX/customizable-functions-templates.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// OK: function template +template <typename T> +custom void f1(T x) { } + +// OK: multiple template parameters +template <typename T, typename U> +custom void f2(T x, U y) { } + +// OK: template with constraints +template <typename T> + requires std::integral<T> +custom void f3(T x) { } + +// OK: abbreviated function template +custom void f4(auto x) { } + +// ERROR: member function template +struct S { + template <typename T> + custom void f5(T x) { } // expected-error {{custom functions cannot be member functions}} +}; + +// OK: template specialization (becomes specialized custom function) +template <> +custom void f1<int>(int x) { } + +// Test instantiation +void test() { + f1(42); // OK + f1(3.14); // OK + f2(1, 2); // OK + f3(5); // OK + f4("hello"); // OK +} +``` + +#### 2.3 Type and Return Type Deduction +**File**: `clang/test/SemaCXX/customizable-functions-deduction.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// OK: explicit return type +custom int f1() { return 42; } + +// OK: auto return type +custom auto f2() { return 42; } + +// OK: decltype(auto) +custom decltype(auto) f3() { return 42; } + +// OK: trailing return type +custom auto f4() -> int { return 42; } + +// OK: template return type deduction +template <typename T> +custom auto f5(T x) { return x; } + +// ERROR: inconsistent return +custom auto f6(bool b) { + if (b) + return 42; // int + else + return 3.14; // expected-error {{return type deduction conflicts}} +} + +// Test return type propagation +void test() { + static_assert(std::is_same_v<decltype(f1()), int>); + static_assert(std::is_same_v<decltype(f2()), int>); + static_assert(std::is_same_v<decltype(f5(42)), int>); +} +``` + +### 3. Code Generation Tests + +#### 3.1 Basic CPO Generation +**File**: `clang/test/CodeGenCXX/customizable-functions-basic-codegen.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -triple x86_64-linux-gnu -emit-llvm %s -o - | FileCheck %s + +custom void simple() { } + +// CHECK: @[[INSTANCE:_ZN10CPO_DETAIL9simple_fnE]] = linkonce_odr global +// CHECK: define {{.*}} @[[IMPL:_ZN10CPO_DETAIL11simple_implEv]]() + +// CHECK: define {{.*}} @[[FALLBACK:.*]]() +// CHECK: call {{.*}} @[[IMPL]]() + +// Verify functor class structure +// CHECK: %[[FUNCTOR:.*]] = type { i8 } +``` + +#### 3.2 Tag Invoke Call +**File**: `clang/test/CodeGenCXX/customizable-functions-tag-invoke.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -triple x86_64-linux-gnu -emit-llvm %s -o - | FileCheck %s + +namespace lib { + custom void process(auto& x) { + x.default_impl(); + } +} + +namespace user { + struct MyType { + void default_impl(); + void custom_impl(); + }; + + // User customization + void tag_invoke(lib::process_fn, MyType& m) { + m.custom_impl(); + } +} + +void test() { + user::MyType obj; + lib::process(obj); +} + +// CHECK: define {{.*}} @_ZN4user10tag_invokeEN3lib10process_fnERNS_6MyTypeE +// CHECK: call {{.*}} @_ZN4user6MyType11custom_implEv +``` + +#### 3.3 Template Instantiation +**File**: `clang/test/CodeGenCXX/customizable-functions-template-codegen.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -triple x86_64-linux-gnu -emit-llvm %s -o - | FileCheck %s + +template <typename T> +custom void print(T const& value) { + // default implementation +} + +void test() { + print(42); + print(3.14); + print("hello"); +} + +// CHECK: define {{.*}} @{{.*}}print{{.*}}i{{.*}} +// CHECK: define {{.*}} @{{.*}}print{{.*}}d{{.*}} +// CHECK: define {{.*}} @{{.*}}print{{.*}}PKc{{.*}} +``` + +### 4. Integration Tests + +#### 4.1 Complete Tag Invoke Workflow +**File**: `clang/test/SemaCXX/customizable-functions-integration.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s +// expected-no-diagnostics + +#include <iostream> +#include <string> + +// Library defines customization point +namespace lib { + custom void serialize(auto const& value, std::ostream& out) { + out << value; // default: use operator<< + } +} + +// User type with customization +namespace user { + struct Person { + std::string name; + int age; + }; + + // User provides custom serialization + void tag_invoke(lib::serialize_fn, Person const& p, std::ostream& out) { + out << "Person{name=" << p.name << ", age=" << p.age << "}"; + } +} + +// Test that it all works +void test() { + // Built-in type uses default implementation + lib::serialize(42, std::cout); + + // User type uses custom implementation + user::Person p{"Alice", 30}; + lib::serialize(p, std::cout); + + // String uses default implementation + lib::serialize(std::string("hello"), std::cout); +} +``` + +#### 4.2 ADL and Overload Resolution +**File**: `clang/test/SemaCXX/customizable-functions-adl.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s +// expected-no-diagnostics + +namespace lib { + custom void process(auto& x) { + x.process(); + } +} + +namespace ns1 { + struct Type1 { + void process() { } + }; + + void tag_invoke(lib::process_fn, Type1& t) { + // Custom implementation for ns1::Type1 + } +} + +namespace ns2 { + struct Type2 { + void process() { } + }; + + void tag_invoke(lib::process_fn, Type2& t) { + // Custom implementation for ns2::Type2 + } +} + +void test() { + ns1::Type1 t1; + ns2::Type2 t2; + + // ADL finds the right tag_invoke in each namespace + lib::process(t1); // calls ns1::tag_invoke + lib::process(t2); // calls ns2::tag_invoke +} +``` + +#### 4.3 Constraints and SFINAE +**File**: `clang/test/SemaCXX/customizable-functions-constraints.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +#include <concepts> + +// Constrained custom function +custom void process_integral(auto x) + requires std::integral<decltype(x)> +{ + // process integral type +} + +namespace user { + struct NotIntegral { }; + + // This should not be selected even with tag_invoke + void tag_invoke(process_integral_fn, NotIntegral x) { } +} + +void test() { + process_integral(42); // OK + process_integral('a'); // OK + process_integral(true); // OK + + user::NotIntegral ni; + process_integral(ni); // expected-error {{no matching function}} + process_integral(3.14); // expected-error {{no matching function}} +} +``` + +### 5. AST Tests + +#### 5.1 AST Structure +**File**: `clang/test/AST/customizable-functions-ast-dump.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -ast-dump %s | FileCheck %s + +custom void test() { } + +// CHECK: FunctionDecl {{.*}} test 'void ()' +// CHECK-NEXT: CompoundStmt +// CHECK: NamespaceDecl {{.*}} CPO_DETAIL +// CHECK-NEXT: FunctionDecl {{.*}} referenced test_impl 'void ()' +// CHECK-NEXT: CXXRecordDecl {{.*}} test_fn +// CHECK-NEXT: CXXMethodDecl {{.*}} operator() 'auto (auto) const' +// CHECK-NEXT: CXXMethodDecl {{.*}} operator() 'auto (auto) const' +// CHECK: VarDecl {{.*}} test 'const test_fn':'const CPO_DETAIL::test_fn' inline constexpr +``` + +#### 5.2 AST Printing +**File**: `clang/test/AST/customizable-functions-ast-print.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -ast-print %s | FileCheck %s + +custom void test() { } + +// CHECK: custom void test() { +// CHECK: } + +// CHECK: namespace CPO_DETAIL { +// CHECK: void test_impl() +// CHECK: struct test_fn { +// CHECK: } +// CHECK: inline constexpr test_fn test; +``` + +### 6. Diagnostic Tests + +#### 6.1 Error Messages +**File**: `clang/test/SemaCXX/customizable-functions-errors.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Test clear error messages + +struct S { + custom void member() { } + // expected-error@-1 {{custom functions cannot be member functions}} + // expected-note@-2 {{remove 'custom' to declare a regular member function}} +}; + +custom void no_body(); +// expected-error@-1 {{custom functions must have a definition}} +// expected-note@-2 {{add a function body}} + +static custom void with_static() { } +// expected-error@-1 {{'custom' cannot be combined with 'static'}} + +custom virtual void with_virtual() { } +// expected-error@-1 {{custom functions cannot be virtual}} +``` + +#### 6.2 Warning Messages +**File**: `clang/test/SemaCXX/customizable-functions-warnings.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Test warnings + +custom void unused() { } +// expected-warning@-1 {{custom function 'unused' is never used}} + +custom void recursive(auto& x) { + recursive(x); // expected-warning {{recursive call to custom function may cause infinite loop}} +} +``` + +### 7. Edge Cases and Corner Cases + +#### 7.1 Name Conflicts +**File**: `clang/test/SemaCXX/customizable-functions-name-conflicts.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +namespace test1 { + custom void func() { } + void func_fn() { } // expected-error {{redefinition of 'func_fn'}} + // expected-note@-3 {{previous definition generated for custom function 'func'}} +} + +namespace test2 { + custom void func() { } + namespace CPO_DETAIL { // expected-error {{redefinition of 'CPO_DETAIL'}} + // expected-note@-2 {{previous definition generated for custom function 'func'}} + } +} +``` + +#### 7.2 Nested Namespaces +**File**: `clang/test/SemaCXX/customizable-functions-nested-namespaces.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s +// expected-no-diagnostics + +namespace outer { + namespace inner { + custom void func() { } + } +} + +void test() { + outer::inner::func(); // OK +} +``` + +#### 7.3 Forward References +**File**: `clang/test/SemaCXX/customizable-functions-forward-refs.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Can we reference the CPO before it's fully defined? +namespace test { + void use_it(); // declares function that uses CPO + + custom void cpo() { } + + void use_it() { + cpo(); // OK + } +} +``` + +### 8. Performance Tests + +#### 8.1 Zero Overhead +**File**: `clang/test/CodeGenCXX/customizable-functions-zero-overhead.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -triple x86_64-linux-gnu -emit-llvm -O2 %s -o - | FileCheck %s + +custom void simple(int& x) { + x++; +} + +void test(int& x) { + simple(x); +} + +// With -O2, should inline to just x++ +// CHECK-LABEL: @_Z4testRi +// CHECK: add nsw i32 {{.*}}, 1 +// CHECK-NEXT: store +// CHECK-NEXT: ret +``` + +#### 8.2 Compile Time +**File**: `clang/test/CodeGenCXX/customizable-functions-compile-time.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -ftime-report %s -o /dev/null 2>&1 | FileCheck %s + +// Many custom functions +custom void f1() { } +custom void f2() { } +custom void f3() { } +// ... many more ... +custom void f100() { } + +// CHECK: Custom Function Transformation +// CHECK: Total Execution Time: {{[0-9.]+}} +``` + +### 9. Module and PCH Tests + +#### 9.1 Modules +**File**: `clang/test/Modules/customizable-functions.cppm` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -emit-module-interface %s -o %t.pcm +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fmodule-file=%t.pcm -verify %s + +export module test; + +export custom void exported_cpo() { } + +// Test that importing works +// RUN: echo 'import test; void use() { exported_cpo(); }' | %clang_cc1 -std=c++20 -fcustomizable-functions -fmodule-file=%t.pcm -x c++ - -fsyntax-only -verify +``` + +#### 9.2 Precompiled Headers +**File**: `clang/test/PCH/customizable-functions.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -emit-pch %s -o %t.pch +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -include-pch %t.pch -verify %s + +#ifndef HEADER +#define HEADER + +custom void in_header() { } + +#else + +void test() { + in_header(); // OK +} + +#endif +``` + +### 10. Standard Library Integration Tests + +#### 10.1 Ranges Integration +**File**: `clang/test/SemaCXX/customizable-functions-ranges.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s +// expected-no-diagnostics + +#include <ranges> +#include <vector> + +namespace lib { + custom void for_each(auto&& range, auto&& func) + requires std::ranges::range<decltype(range)> + { + for (auto&& elem : range) { + func(elem); + } + } +} + +void test() { + std::vector<int> v = {1, 2, 3}; + lib::for_each(v, [](int x) { }); // OK +} +``` + +### 11. Regression Tests + +Tests for specific bugs found during development. + +**File**: `clang/test/SemaCXX/customizable-functions-regression.cpp` + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s +// expected-no-diagnostics + +// Add regression tests as bugs are found and fixed +``` + +--- + +## Test Execution Strategy + +### Continuous Testing + +1. **Pre-commit**: Run parser and semantic tests +2. **Post-commit**: Run full test suite including codegen +3. **Nightly**: Run performance benchmarks +4. **Release**: Run extensive integration tests + +### Test Coverage Goals + +- **Parser**: 100% coverage of syntax cases +- **Sema**: 100% coverage of semantic rules +- **CodeGen**: 90%+ coverage of generation patterns +- **Integration**: Cover all major use cases + +### Performance Benchmarks + +Create benchmarks comparing: +1. Manual CPO implementation vs. generated +2. Compilation time impact +3. Runtime performance (should be identical) +4. Binary size impact + +### Test Infrastructure + +**Tools**: +- `FileCheck` for verifying compiler output +- `not` for verifying errors +- `diff` for comparing generated code +- Custom lit tests for end-to-end scenarios + +**Test Organization**: +``` +clang/test/ +├── Parser/ +│ ├── cxx-customizable-functions-*.cpp +├── SemaCXX/ +│ ├── customizable-functions-*.cpp +├── CodeGenCXX/ +│ ├── customizable-functions-*.cpp +├── AST/ +│ ├── customizable-functions-*.cpp +├── Modules/ +│ ├── customizable-functions.cppm +└── PCH/ + ├── customizable-functions.cpp +``` + +--- + +## Acceptance Criteria + +Before merging, the feature must: + +1. **Pass all tests**: 100% test pass rate +2. **No regressions**: All existing tests still pass +3. **Performance neutral**: No measurable slowdown in non-custom code +4. **Documentation**: Complete user documentation +5. **Code review**: Approved by LLVM community +6. **Examples**: Working examples for common use cases + +--- + +## Future Test Considerations + +Tests to add in future phases: + +1. **Member function support** (if added) +2. **Attributes and customization options** +3. **TInCuP library integration** +4. **Cross-platform testing** (Windows, macOS, Linux) +5. **Debugger integration** (LLDB, GDB) +6. **Static analyzer support** +7. **Code completion and IDE integration** + +--- + +## Test Metrics + +Track these metrics during development: + +- **Test count**: Number of test cases +- **Code coverage**: Percentage of new code covered +- **Bug count**: Bugs found via testing +- **Regression count**: Regressions caught +- **Performance**: Compile time impact + +Target: 90%+ code coverage, 0 known regressions before release. >From 667fc38cb1339f4ce9c9136c1637eec6898331fc Mon Sep 17 00:00:00 2001 From: Claude <[email protected]> Date: Mon, 17 Nov 2025 18:01:54 +0000 Subject: [PATCH 2/2] Add build script and development guide for customizable functions This commit adds development tooling to streamline building and testing the customizable functions feature. Added files: - custom-functions-dev.sh: Comprehensive build script with commands for: * CMake configuration (debug/release/minimal modes) * Building clang with optimized settings * Running tests (by category or pattern) * Build management (clean, rebuild, info) * Fast iteration workflow for development - CUSTOMIZABLE_FUNCTIONS_DEV.md: Complete development guide including: * Quick start instructions * Development workflow examples * Build optimization details * IDE integration setup * Testing strategy * Troubleshooting tips * Project structure overview The build script optimizes for fast development iteration: - Ninja build system for parallel builds - Single target (X86) to reduce build time - Optimized TableGen for faster generation - Split DWARF for faster linking - Easy test filtering by category Example usage: ./custom-functions-dev.sh configure debug ./custom-functions-dev.sh build ./custom-functions-dev.sh test parser --- CUSTOMIZABLE_FUNCTIONS_DEV.md | 392 +++++++++++++++++++++++++++++++ custom-functions-dev.sh | 418 ++++++++++++++++++++++++++++++++++ 2 files changed, 810 insertions(+) create mode 100644 CUSTOMIZABLE_FUNCTIONS_DEV.md create mode 100755 custom-functions-dev.sh diff --git a/CUSTOMIZABLE_FUNCTIONS_DEV.md b/CUSTOMIZABLE_FUNCTIONS_DEV.md new file mode 100644 index 0000000000000..088d403cf0960 --- /dev/null +++ b/CUSTOMIZABLE_FUNCTIONS_DEV.md @@ -0,0 +1,392 @@ +# Customizable Functions Feature Development Guide + +This guide covers developing and testing the Customizable Functions feature in Clang. + +## Quick Start + +### First Time Setup + +```bash +# 1. Configure the build (debug mode, optimized for development) +./custom-functions-dev.sh configure debug + +# 2. Build clang +./custom-functions-dev.sh build + +# 3. Verify the build +./custom-functions-dev.sh info +``` + +### Development Workflow + +```bash +# Make changes to source files... + +# Build only what changed +./custom-functions-dev.sh build + +# Run customizable functions tests +./custom-functions-dev.sh test customizable + +# Or run specific test categories +./custom-functions-dev.sh test parser +./custom-functions-dev.sh test sema +./custom-functions-dev.sh test codegen +``` + +## Build Script Commands + +### Configure + +```bash +# Debug build (default - best for development) +./custom-functions-dev.sh configure debug + +# Release build (for performance testing) +./custom-functions-dev.sh configure release + +# Minimal build (fastest iteration) +./custom-functions-dev.sh configure minimal +``` + +### Build + +```bash +# Build clang +./custom-functions-dev.sh build + +# Build specific target +./custom-functions-dev.sh build check-clang-sema + +# Build with custom job count +BUILD_JOBS=8 ./custom-functions-dev.sh build +``` + +### Test + +```bash +# Run all customizable functions tests +./custom-functions-dev.sh test customizable + +# Run parser tests +./custom-functions-dev.sh test parser + +# Run semantic analysis tests +./custom-functions-dev.sh test sema + +# Run code generation tests +./custom-functions-dev.sh test codegen + +# Run AST tests +./custom-functions-dev.sh test ast + +# Run all Clang tests +./custom-functions-dev.sh test all + +# Run tests matching specific pattern +./custom-functions-dev.sh test "custom.*syntax" +``` + +### Utility Commands + +```bash +# Show build information +./custom-functions-dev.sh info + +# Clean build directory +./custom-functions-dev.sh clean + +# Rebuild from scratch +./custom-functions-dev.sh rebuild + +# Show help +./custom-functions-dev.sh help +``` + +## Build Optimization + +The build script uses several optimizations for faster development: + +- **Ninja build system**: Parallel builds with dependency tracking +- **Optimized TableGen**: Faster build times +- **Split DWARF**: Faster linking with debug info +- **Single target (X86)**: Reduces build time +- **Minimal LLVM projects**: Only builds Clang, not other tools + +### Build Times (Approximate) + +On a typical development machine (8 cores): + +- **Initial build**: 20-40 minutes (full Clang build) +- **Incremental parser change**: 30-60 seconds +- **Incremental Sema change**: 1-2 minutes +- **Incremental test-only**: 5-10 seconds + +## Development Tips + +### Fast Iteration Cycle + +1. **Use incremental builds**: The script automatically detects changes +2. **Test specific categories**: Don't run all tests every time +3. **Use minimal build mode**: For rapid prototyping + +```bash +# Example fast iteration +vim clang/lib/Parse/ParseDecl.cpp +./custom-functions-dev.sh build +./custom-functions-dev.sh test parser +``` + +### Debugging Build Issues + +```bash +# Show detailed build information +./custom-functions-dev.sh info + +# Clean and rebuild +./custom-functions-dev.sh rebuild debug + +# Verbose build output +cd build-custom-functions && ninja -v clang +``` + +### Running Individual Tests + +```bash +# Build directory contains lit tool +cd build-custom-functions + +# Run a specific test file +./bin/llvm-lit -v ../clang/test/Parser/cxx-customizable-functions.cpp + +# Run with verbose output +./bin/llvm-lit -v -a ../clang/test/SemaCXX/customizable-functions-*.cpp +``` + +### Using the Built Clang + +```bash +# Direct path to built clang +./build-custom-functions/bin/clang --version + +# Test the custom keyword (when implemented) +./build-custom-functions/bin/clang -std=c++20 -fcustomizable-functions -fsyntax-only test.cpp + +# See generated AST +./build-custom-functions/bin/clang -std=c++20 -fcustomizable-functions -Xclang -ast-dump test.cpp +``` + +## Project Structure + +``` +llvm-project/ +├── custom-functions-dev.sh # Main build script +├── CUSTOMIZABLE_FUNCTIONS_DEV.md # This file +├── build-custom-functions/ # Build output directory +│ ├── bin/clang # Built clang binary +│ └── compile_commands.json # For IDE integration +├── clang/ +│ ├── docs/ +│ │ ├── CustomizableFunctionsDesign.md # Design document +│ │ └── CustomizableFunctionsTestPlan.md # Test plan +│ ├── include/clang/ +│ │ ├── Basic/ +│ │ │ ├── TokenKinds.def # Add 'custom' keyword here +│ │ │ └── LangOptions.def # Add language option here +│ │ ├── Parse/ +│ │ │ └── Parser.h # Parser interface +│ │ ├── Sema/ +│ │ │ ├── Sema.h # Semantic analysis interface +│ │ │ └── SemaCustomFunction.h # Custom function transformation (new) +│ │ └── AST/ +│ │ └── Decl.h # AST node declarations +│ ├── lib/ +│ │ ├── Parse/ +│ │ │ └── ParseDecl.cpp # Parse 'custom' keyword +│ │ ├── Sema/ +│ │ │ ├── SemaDecl.cpp # Semantic analysis +│ │ │ └── SemaCustomFunction.cpp # Transform logic (new) +│ │ └── AST/ +│ │ └── Decl.cpp # AST implementation +│ └── test/ +│ ├── Parser/ +│ │ └── cxx-customizable-functions-*.cpp +│ ├── SemaCXX/ +│ │ └── customizable-functions-*.cpp +│ ├── CodeGenCXX/ +│ │ └── customizable-functions-*.cpp +│ └── AST/ +│ └── customizable-functions-*.cpp +``` + +## Common Development Tasks + +### Adding a New Keyword + +1. Add to `clang/include/clang/Basic/TokenKinds.def` +2. Rebuild: `./custom-functions-dev.sh build` +3. Test: `./custom-functions-dev.sh test parser` + +### Adding Parser Support + +1. Modify `clang/lib/Parse/ParseDecl.cpp` +2. Add tests in `clang/test/Parser/` +3. Build and test: + ```bash + ./custom-functions-dev.sh build + ./custom-functions-dev.sh test parser + ``` + +### Adding Semantic Analysis + +1. Create `clang/lib/Sema/SemaCustomFunction.cpp` +2. Add hook in `clang/lib/Sema/SemaDecl.cpp` +3. Add tests in `clang/test/SemaCXX/` +4. Build and test: + ```bash + ./custom-functions-dev.sh build + ./custom-functions-dev.sh test sema + ``` + +### Adding Code Generation + +1. Modify `clang/lib/CodeGen/CGDecl.cpp` +2. Add tests in `clang/test/CodeGenCXX/` +3. Build and test: + ```bash + ./custom-functions-dev.sh build + ./custom-functions-dev.sh test codegen + ``` + +## IDE Integration + +### Compile Commands (for clangd, CLion, etc.) + +The build script automatically creates a symlink to `compile_commands.json`: + +```bash +llvm-project/compile_commands.json -> build-custom-functions/compile_commands.json +``` + +This enables IDE features like: +- Code completion +- Jump to definition +- Error highlighting +- Refactoring tools + +### VS Code + +Install the clangd extension and it will automatically find the compile commands. + +### CLion + +CLion will detect the CMake project automatically. Point it to `build-custom-functions/`. + +## Testing Strategy + +### Test Categories + +1. **Parser Tests**: Syntax validation + - `./custom-functions-dev.sh test parser` + +2. **Semantic Tests**: Type checking, constraints + - `./custom-functions-dev.sh test sema` + +3. **CodeGen Tests**: LLVM IR generation + - `./custom-functions-dev.sh test codegen` + +4. **AST Tests**: AST structure verification + - `./custom-functions-dev.sh test ast` + +5. **Integration Tests**: End-to-end workflows + - `./custom-functions-dev.sh test customizable` + +### Writing New Tests + +Create test files following this pattern: + +```cpp +// RUN: %clang_cc1 -std=c++20 -fcustomizable-functions -fsyntax-only -verify %s + +// Test case description +custom void my_test() { } // expected-no-diagnostics + +// Or with expected errors +custom void bad_test(); // expected-error {{custom functions must have a body}} +``` + +Place in appropriate directory: +- Parser tests: `clang/test/Parser/` +- Semantic tests: `clang/test/SemaCXX/` +- CodeGen tests: `clang/test/CodeGenCXX/` + +## Troubleshooting + +### Build Fails + +```bash +# Clean and rebuild +./custom-functions-dev.sh rebuild + +# Check for CMake issues +cd build-custom-functions +cmake .. + +# Verbose build to see errors +ninja -v clang +``` + +### Tests Fail + +```bash +# Run specific test with verbose output +cd build-custom-functions +./bin/llvm-lit -v ../clang/test/Parser/cxx-customizable-functions.cpp -a + +# Check test expectations +cat ../clang/test/Parser/cxx-customizable-functions.cpp +``` + +### Performance Issues + +```bash +# Use release build for performance testing +./custom-functions-dev.sh configure release +./custom-functions-dev.sh build + +# Increase parallel jobs (if you have RAM) +BUILD_JOBS=16 ./custom-functions-dev.sh build +``` + +## Environment Variables + +- `BUILD_JOBS`: Number of parallel build jobs (default: nproc) +- `BUILD_TYPE`: Override build type (Debug/Release) +- `CC`: C compiler to use +- `CXX`: C++ compiler to use + +Example: +```bash +export BUILD_JOBS=12 +export CC=clang +export CXX=clang++ +./custom-functions-dev.sh configure debug +``` + +## References + +- [Design Document](clang/docs/CustomizableFunctionsDesign.md) +- [Test Plan](clang/docs/CustomizableFunctionsTestPlan.md) +- [LLVM Testing Infrastructure](https://llvm.org/docs/TestingGuide.html) +- [Clang Internals Manual](https://clang.llvm.org/docs/InternalsManual.html) + +## Next Steps + +Once the build is configured: + +1. Review the design documents +2. Start with Phase 1: Add keyword and basic parsing +3. Add tests incrementally +4. Iterate quickly with the build script + +Happy coding! diff --git a/custom-functions-dev.sh b/custom-functions-dev.sh new file mode 100755 index 0000000000000..3ce56a4f5044b --- /dev/null +++ b/custom-functions-dev.sh @@ -0,0 +1,418 @@ +#!/bin/bash +################################################################################ +# Build Script for Customizable Functions Feature Development +# +# This script configures, builds, and tests the customizable functions feature +# in Clang with optimized settings for faster iteration. +# +# Usage: +# ./build-customizable-functions.sh [command] [options] +# +# Commands: +# configure [debug|release|minimal] - Configure CMake build +# build [target] - Build clang (or specific target) +# test [pattern] - Run tests (optionally filter by pattern) +# clean - Clean build directory +# rebuild - Clean and rebuild +# help - Show this help +# +# Examples: +# ./build-customizable-functions.sh configure debug +# ./build-customizable-functions.sh build +# ./build-customizable-functions.sh test customizable +# ./build-customizable-functions.sh rebuild +################################################################################ + +set -e # Exit on error + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Directories +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LLVM_DIR="${SCRIPT_DIR}" +BUILD_DIR="${SCRIPT_DIR}/build-custom-functions" +INSTALL_DIR="${BUILD_DIR}/install" + +# Build settings +DEFAULT_BUILD_TYPE="Debug" +NUM_JOBS=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4) + +################################################################################ +# Helper Functions +################################################################################ + +print_header() { + echo -e "${BLUE}======================================${NC}" + echo -e "${BLUE}$1${NC}" + echo -e "${BLUE}======================================${NC}" +} + +print_success() { + echo -e "${GREEN}✓ $1${NC}" +} + +print_error() { + echo -e "${RED}✗ $1${NC}" +} + +print_warning() { + echo -e "${YELLOW}⚠ $1${NC}" +} + +print_info() { + echo -e "${BLUE}ℹ $1${NC}" +} + +show_help() { + cat << EOF +${BLUE}Customizable Functions Build Script${NC} + +${GREEN}Usage:${NC} + ./build-customizable-functions.sh [command] [options] + +${GREEN}Commands:${NC} + ${YELLOW}configure [mode]${NC} Configure CMake build + Modes: debug (default), release, minimal + ${YELLOW}build [target]${NC} Build clang or specific target + Targets: clang, check-clang, check-clang-sema + ${YELLOW}test [pattern]${NC} Run tests, optionally filter by pattern + Examples: customizable, parser, sema + ${YELLOW}clean${NC} Clean build directory + ${YELLOW}rebuild${NC} Clean and rebuild from scratch + ${YELLOW}info${NC} Show build information + ${YELLOW}help${NC} Show this help + +${GREEN}Examples:${NC} + # Initial setup - configure and build + ./build-customizable-functions.sh configure debug + ./build-customizable-functions.sh build + + # Run all customizable functions tests + ./build-customizable-functions.sh test customizable + + # Run only parser tests + ./build-customizable-functions.sh test parser + + # Rebuild everything from scratch + ./build-customizable-functions.sh rebuild + + # Build and run specific test + ./build-customizable-functions.sh build check-clang + +${GREEN}Build Modes:${NC} + ${YELLOW}debug${NC} - Debug build with assertions (default) + - Best for development and debugging + - Slower but easier to debug + + ${YELLOW}release${NC} - Optimized release build + - Faster but harder to debug + - Use for performance testing + + ${YELLOW}minimal${NC} - Minimal debug build + - Only builds clang, not all of LLVM + - Fastest iteration time + +${GREEN}Environment Variables:${NC} + ${YELLOW}BUILD_JOBS${NC} Number of parallel build jobs (default: ${NUM_JOBS}) + ${YELLOW}BUILD_TYPE${NC} Build type override (Debug/Release/RelWithDebInfo) + ${YELLOW}CC${NC} C compiler (default: auto-detect) + ${YELLOW}CXX${NC} C++ compiler (default: auto-detect) + +EOF +} + +################################################################################ +# CMake Configuration +################################################################################ + +configure_build() { + local build_mode="${1:-debug}" + + print_header "Configuring CMake Build (${build_mode} mode)" + + # Create build directory + mkdir -p "${BUILD_DIR}" + cd "${BUILD_DIR}" + + # Determine build type + local cmake_build_type="${DEFAULT_BUILD_TYPE}" + local extra_flags="" + + case "${build_mode}" in + debug) + cmake_build_type="Debug" + print_info "Debug build: Assertions enabled, optimizations disabled" + ;; + release) + cmake_build_type="Release" + print_info "Release build: Optimizations enabled, assertions disabled" + ;; + minimal) + cmake_build_type="Debug" + extra_flags="-DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_ASSERTIONS=ON" + print_info "Minimal build: Only X86 target, debug mode" + ;; + *) + print_error "Unknown build mode: ${build_mode}" + print_info "Valid modes: debug, release, minimal" + exit 1 + ;; + esac + + # Allow override via environment + if [ -n "${BUILD_TYPE}" ]; then + cmake_build_type="${BUILD_TYPE}" + print_warning "Build type overridden to: ${cmake_build_type}" + fi + + print_info "Build directory: ${BUILD_DIR}" + print_info "Install directory: ${INSTALL_DIR}" + print_info "Build type: ${cmake_build_type}" + print_info "Parallel jobs: ${NUM_JOBS}" + + # Configure CMake with optimized settings for development + cmake -G Ninja \ + -DCMAKE_BUILD_TYPE="${cmake_build_type}" \ + -DCMAKE_INSTALL_PREFIX="${INSTALL_DIR}" \ + -DLLVM_ENABLE_PROJECTS="clang" \ + -DLLVM_ENABLE_RUNTIMES="" \ + -DLLVM_TARGETS_TO_BUILD="X86" \ + -DLLVM_INCLUDE_TESTS=ON \ + -DLLVM_INCLUDE_EXAMPLES=OFF \ + -DLLVM_INCLUDE_DOCS=OFF \ + -DLLVM_ENABLE_ASSERTIONS=ON \ + -DLLVM_ENABLE_WERROR=OFF \ + -DLLVM_OPTIMIZED_TABLEGEN=ON \ + -DLLVM_USE_SPLIT_DWARF=ON \ + -DCLANG_ENABLE_STATIC_ANALYZER=ON \ + -DCLANG_ENABLE_ARCMT=OFF \ + -DCLANG_BUILD_EXAMPLES=OFF \ + -DLLVM_BUILD_LLVM_DYLIB=OFF \ + -DLLVM_LINK_LLVM_DYLIB=OFF \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ + ${extra_flags} \ + "${LLVM_DIR}/llvm" + + print_success "CMake configuration complete" + print_info "Build files generated in: ${BUILD_DIR}" + + # Create symlink to compile_commands.json for IDE support + if [ -f "${BUILD_DIR}/compile_commands.json" ]; then + ln -sf "${BUILD_DIR}/compile_commands.json" "${LLVM_DIR}/compile_commands.json" + print_success "Created compile_commands.json symlink for IDE support" + fi +} + +################################################################################ +# Build Functions +################################################################################ + +build_target() { + local target="${1:-clang}" + + if [ ! -d "${BUILD_DIR}" ]; then + print_error "Build directory not found. Run 'configure' first." + exit 1 + fi + + print_header "Building ${target}" + + cd "${BUILD_DIR}" + + # Use BUILD_JOBS env var if set + local jobs="${BUILD_JOBS:-${NUM_JOBS}}" + + print_info "Building with ${jobs} parallel jobs..." + + local start_time=$(date +%s) + + if ninja -j "${jobs}" "${target}"; then + local end_time=$(date +%s) + local duration=$((end_time - start_time)) + print_success "Build complete in ${duration} seconds" + + if [ "${target}" = "clang" ]; then + print_info "Clang binary: ${BUILD_DIR}/bin/clang" + fi + else + print_error "Build failed" + exit 1 + fi +} + +################################################################################ +# Test Functions +################################################################################ + +run_tests() { + local pattern="${1:-customizable-functions}" + + if [ ! -d "${BUILD_DIR}" ]; then + print_error "Build directory not found. Run 'configure' first." + exit 1 + fi + + cd "${BUILD_DIR}" + + print_header "Running Tests" + print_info "Filter pattern: ${pattern}" + + # Check if pattern is a specific test category + case "${pattern}" in + customizable*|custom*) + print_info "Running customizable functions tests..." + ninja check-clang 2>&1 | grep -i "customizable" || true + ;; + parser) + print_info "Running parser tests..." + ./bin/llvm-lit -v "${LLVM_DIR}/clang/test/Parser" -a 2>&1 | grep -i "customizable" || true + ;; + sema*) + print_info "Running semantic analysis tests..." + ./bin/llvm-lit -v "${LLVM_DIR}/clang/test/SemaCXX" -a 2>&1 | grep -i "customizable" || true + ;; + codegen) + print_info "Running code generation tests..." + ./bin/llvm-lit -v "${LLVM_DIR}/clang/test/CodeGenCXX" -a 2>&1 | grep -i "customizable" || true + ;; + ast) + print_info "Running AST tests..." + ./bin/llvm-lit -v "${LLVM_DIR}/clang/test/AST" -a 2>&1 | grep -i "customizable" || true + ;; + all) + print_info "Running all Clang tests..." + ninja check-clang + ;; + *) + print_info "Running tests matching: ${pattern}" + ./bin/llvm-lit -v "${LLVM_DIR}/clang/test" --filter="${pattern}" + ;; + esac + + print_success "Test run complete" +} + +################################################################################ +# Utility Functions +################################################################################ + +clean_build() { + print_header "Cleaning Build Directory" + + if [ -d "${BUILD_DIR}" ]; then + print_warning "Removing ${BUILD_DIR}..." + rm -rf "${BUILD_DIR}" + print_success "Build directory cleaned" + else + print_info "Build directory does not exist, nothing to clean" + fi +} + +rebuild_all() { + print_header "Rebuilding from Scratch" + + clean_build + configure_build "${1:-debug}" + build_target "clang" + + print_success "Rebuild complete" +} + +show_info() { + print_header "Build Information" + + if [ -d "${BUILD_DIR}" ]; then + echo -e "${GREEN}Build Directory:${NC} ${BUILD_DIR}" + + if [ -f "${BUILD_DIR}/CMakeCache.txt" ]; then + local build_type=$(grep "CMAKE_BUILD_TYPE:" "${BUILD_DIR}/CMakeCache.txt" | cut -d'=' -f2) + echo -e "${GREEN}Build Type:${NC} ${build_type}" + + local targets=$(grep "LLVM_TARGETS_TO_BUILD:" "${BUILD_DIR}/CMakeCache.txt" | cut -d'=' -f2) + echo -e "${GREEN}Targets:${NC} ${targets}" + fi + + if [ -f "${BUILD_DIR}/bin/clang" ]; then + echo -e "${GREEN}Clang Binary:${NC} ${BUILD_DIR}/bin/clang" + echo -e "${GREEN}Clang Version:${NC}" + "${BUILD_DIR}/bin/clang" --version | head -1 + else + echo -e "${YELLOW}Clang not built yet${NC}" + fi + + echo "" + echo -e "${GREEN}Available Test Commands:${NC}" + echo -e " ./build-customizable-functions.sh test customizable ${BLUE}# Run customizable functions tests${NC}" + echo -e " ./build-customizable-functions.sh test parser ${BLUE}# Run parser tests${NC}" + echo -e " ./build-customizable-functions.sh test sema ${BLUE}# Run semantic tests${NC}" + echo -e " ./build-customizable-functions.sh test codegen ${BLUE}# Run codegen tests${NC}" + echo -e " ./build-customizable-functions.sh test all ${BLUE}# Run all tests${NC}" + else + print_warning "Build not configured yet" + echo -e "Run: ${YELLOW}./build-customizable-functions.sh configure${NC}" + fi +} + +################################################################################ +# Quick Development Workflow Commands +################################################################################ + +quick_test() { + print_header "Quick Test Workflow" + print_info "Building and testing customizable functions..." + + build_target "clang" + run_tests "customizable" + + print_success "Quick test complete" +} + +################################################################################ +# Main Script +################################################################################ + +main() { + local command="${1:-help}" + shift || true + + case "${command}" in + configure|config) + configure_build "$@" + ;; + build) + build_target "$@" + ;; + test) + run_tests "$@" + ;; + clean) + clean_build + ;; + rebuild) + rebuild_all "$@" + ;; + info) + show_info + ;; + quick) + quick_test + ;; + help|--help|-h) + show_help + ;; + *) + print_error "Unknown command: ${command}" + echo "" + show_help + exit 1 + ;; + esac +} + +# Run main function +main "$@" _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
