Re: [EXT] Duplicate visits to expressions defined in RHS of a setter
Thank you Eric and Paul. I used line/col numbers to prevent duplicates.
This probably is good enough for now
To answer some questions, Eric, I am not able to do this before instruction
selection because I want the static compile to finish its resolutions and I
use the metadata gathered in that stage to inform my data collector about
types, methods, etc used. Paul, I am essentially gathering the results of
the static compile into a dictionary that I use to simplify my server side
compile. The product is a hosted DSL that allows developers to integrate
with their Java data models. The first compile collects all information
about the data models into a dictionary, the second compile uses this
dictionary and converts the groovy script into wire format manipulations
instead of pojo gets/set. At runtime the bytecode is exclusively run as a
wire format manipulator without any of the original dependencies. So I have
a need to understand names of properties set/read, methods called etc ... I
would love your (and other groovy gurus) feed back on some of this, but I
am not sure if you will have the time or means to connect
regards
Saravanan
On Tue, Apr 4, 2023 at 2:02 AM Paul King wrote:
> Good suggestions from Eric. Another is that you can "drop some
> breadcrumbs", i.e. place a "seen" node metadata marker and skip the next
> time around. This falls into the category of a workaround rather than a fix
> for your expectation, but we do legitimately walk the tree multiple times
> in a few places. We make sure our transforms are idempotent in that case.
> It doesn't mean there might not be a bug somewhere and we walk more than
> needed for some edge case. Perhaps you could expand your example and we can
> check further.
>
> Cheers, Paul.
>
> On Mon, Apr 3, 2023 at 11:23 PM Milles, Eric (TR Technology) via dev <
> dev@groovy.apache.org> wrote:
>
>> You can try visiting at an earlier compile phase (maybe before
>> instruction selection) to see the AST before some SC transforms are
>> applied. Otherwise, I have used the source location as a key to understand
>> if something is coming in again. Often generated methods will include some
>> of the source elements again.
>>
>>
>>
>> Here is an example of an AST visitor that deals with a number of the
>> idiosyncrasies of the tree.
>> https://github.com/groovy/groovy-eclipse/blob/master/base/org.eclipse.jdt.groovy.core/src/org/eclipse/jdt/groovy/core/util/DepthFirstVisitor.java
>>
>>
>>
>> *From:* Saravanan Palanichamy
>> *Sent:* Saturday, April 1, 2023 1:35 AM
>> *To:* dev@groovy.apache.org
>> *Subject:* [EXT] Duplicate visits to expressions defined in RHS of a
>> setter
>>
>>
>>
>> *External Email:* Use caution with links and attachments.
>>
>>
>>
>> Hello Groovy Devs
>>
>>- I have a class defined MyTestClass in java/kotlin (does not matter)
>>- I have a groovy class defined MyClassDefinedInGroovy
>>- I have code in a groovy function that is creating these classes and
>>setting a variable in the object. I compile this with static compilation
>>turned on
>>- I visit this class using a simple visitor that prints all constants
>>visited (I have attached both code and groovy file in this email)
>>
>> For the class defined in java
>>
>>- In 3.0.16, all constants are visited twice. This is not what I
>>would expect but at least it is consistent
>>- In 3.0.5, the setters with a property based LHS are visited twice,
>>but those with a variable based LHS, only one visit as expected
>>
>> For the class defined in Groovy, In both 3.0.5 and 3.0.16, all constants
>> are visited only once. This is what I would expect
>>
>>
>>
>> Multiple visits to the same code is causing an issue for me because I
>> collect metadata about the code for use elsewhere and I end up getting
>> duplicates that cannot be cleanly ignored (because its not just variables,
>> its anything on the RHS, closures, method calls, etc)
>>
>>
>>
>> I debugged this a bit and found out that for classes imported from Java,
>> the compiler seems to be using PoppingMethodCallExpression and
>> PoppingListOfExpressionsExpression. The latter has code that initializes
>> the parent with two expressions, the tmp variable and the method call. The
>> method call also includes the tmp variable which I think is causing this
>> duplicate visit.
>>
>>
>>
>> public PoppingListOfExpressionsExpression(final TemporaryVariableExpression
>> tmp, final PoppingMethodCallExpression call) {
>>
>> // This array initialization with tmp and call is a problem because call
>> also holds tmp and visits it
>>
>> super(Arrays.asList(tmp, call));
>> this.tmp = tmp;
>> this.call = call;
>> }
>>
>>
>>
>> What are my options if I dont want to handle the duplicate visit? This
>> problem exists in some scenarios in 3.0.5 and is consistently a problem in
>> 3.0.16
>>
>>
>>
>> regards
>>
>> Saravanan
>>
>
Duplicate visits to expressions defined in RHS of a setter
Hello Groovy Devs
- I have a class defined MyTestClass in java/kotlin (does not matter)
- I have a groovy class defined MyClassDefinedInGroovy
- I have code in a groovy function that is creating these classes and
setting a variable in the object. I compile this with static compilation
turned on
- I visit this class using a simple visitor that prints all constants
visited (I have attached both code and groovy file in this email)
For the class defined in java
- In 3.0.16, all constants are visited twice. This is not what I would
expect but at least it is consistent
- In 3.0.5, the setters with a property based LHS are visited twice, but
those with a variable based LHS, only one visit as expected
For the class defined in Groovy, In both 3.0.5 and 3.0.16, all constants
are visited only once. This is what I would expect
Multiple visits to the same code is causing an issue for me because I
collect metadata about the code for use elsewhere and I end up getting
duplicates that cannot be cleanly ignored (because its not just variables,
its anything on the RHS, closures, method calls, etc)
I debugged this a bit and found out that for classes imported from Java,
the compiler seems to be using PoppingMethodCallExpression and
PoppingListOfExpressionsExpression. The latter has code that initializes
the parent with two expressions, the tmp variable and the method call. The
method call also includes the tmp variable which I think is causing this
duplicate visit.
>
> public PoppingListOfExpressionsExpression(final TemporaryVariableExpression
> tmp, final PoppingMethodCallExpression call) {
>
> // This array initialization with tmp and call is a problem because call
> also holds tmp and visits it
>
> super(Arrays.asList(tmp, call));
> this.tmp = tmp;
> this.call = call;
> }
>
>
What are my options if I dont want to handle the duplicate visit? This
problem exists in some scenarios in 3.0.5 and is consistently a problem in
3.0.16
regards
Saravanan
Test.flow
Description: Binary data
TestPopper.kt
Description: Binary data
Re: Disabling auto inferencing
Thanks Jochen and Christopher for helping clarify behaviour. So if I wanted to make sure declaration and not flow type is honored, the quickest way is to make sure flow type matches declaration by casting it? Are there other ways inside the compiler to fix this (some options or some indications in the variable declaration statement itself) (I looked and didnt find anything, but thought I'd ask)? I am using 3.0.5 and I compile statically into the file system regards Saravanan On Sun, Dec 4, 2022 at 12:25 AM Jochen Theodorou wrote: > On 03.12.22 15:50, Christopher Smith wrote: > > I believe the feature at play is "flow typing", and it surprises me that > > it would apply to variables declared with an explicit type. What version > > of Groovy are you targeting, and is this compiled statically or > > dynamically? > > The declaration type is to be understood as an base type for the > variable, not the flow type. Which means in > > Y foo = bar() > foo.x() > > the flow type of foo is whatever is inferred as result type for the call > bar(). The assignment to foo is valid if the result type for the call to > bar() is assignable to Y. The call x() on foo is valid if the method x() > exists on the flow type. > > This is the intended behaviour. > > bye Jochen > >
Disabling auto inferencing
Hello developers
I am developing a DSL and ran into this issue. Groovy auto infers that b is
really a derived clazz and allows me to call the derived clazz method. How
do I prevent this behavior from happening? Are there some settings in the
DeclarationExpression or VariableExpression AST models that will allow me
to prevent this and cause a compile error? (I am writing a compiler
customizer and so have access to do this)
void myMethod() {
> BaseClazz b = new DerivedClazz()
> b.callMethodDefinedOnlyInDerivedClazz("")
> }
>
>
Alternately, this seems to work correctly. But I'd rather not do it this
way if possible
void myMethod() {
> BaseClazz b = (BaseClazz) new DerivedClazz()
> b.callMethodDefinedOnlyInDerivedClazz("")
> }
>
> regards
Saravanan
Unable to breakpoint inside closures generated during AST transformation
Hello folks
In my AST transformation that takes a block of code and executes it as a
closure, I am unable to set a breakpoint inside the block of code at runtime.
This however works if the closure was written as part of the script directly.
What am I missing?
Sample Groovy code
class MyClass {
void method() {
String b = ""
throw new WTHException()
}
}
My invoker
class InvokeClosure {
static ClassNode NODE = ClassHelper.makeNode(InvokeClosure.class)
static void invokeClosure(Closure closure) {
closure.call()
}
}
My AST transformation
void visitMethod(MethodNode methodNode) {
ClosureExpression closureExpression =
GeneralUtils.closureX(methodNode.getCode())
// NODE is a
StaticMethodCallExpression methodExpression =
GeneralUtils.callX(InvokeClosure.NODE, "invokeClosure",
GeneralUtils.args(closure))
methodNode.setCode(GeneralUtils.stmt(methodExpression))
}
When I use this AST transformation, it compiles fine, invokeClosure gets called
when I invoke my method, but I am not able to set a breakpoint at the method()
function in line 1. The IDE disables my breakpoint on execution. What am I
missing in my transformation? I tried setting positions for my closure, and my
statement. Still did not work. Thank you for your responses
regards
Local variable annotations for multi declaration expressions
Hello How do I annotate local variables in a multi declaration? @MyTag def (var1, var2) = ["", ""] --> this works and I am able to see the annotations in my AST def (@MyTag var1, @MyOtherTag var2) = ["", ""] --> This does not work. In the second declaration model, The IDE calls out an error, the compiler is silent with no errors and the AST does not show any annotations as part of the declarationExpression. What is the right way to annotate variables that are part of a multi declaration expression statement? regards Saravanan
Re: Memory leak with org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry
Hello
Any help with this is super appreciated
regards
Saravanan
On 2020/09/08 01:25:11, Saravanan Palanichamy wrote:
> Upon further analysis, if I call
> InvokerHelper.removeClass(clazz)
> after I instantiate and use the instance, then things get unloaded correctly.
> What is the right way to fix this? Is it to call this invoker method for all
> classes I load or is the injected constructor code the problem?
>
> regards
> Saravanan
>
> On 2020/09/08 00:59:34, Saravanan Palanichamy wrote:
> > Hello
> >
> > I am running into a memory leak issue with Groovy. I am not sure this has
> > been fixed in newer releases. I am on 2.5.3
> >
> > * I compile my groovy files into a jar at compile time (I do not load
> > dynamically)
> > * I then use this code to load my jar
> > val classloader =
> > URLClassLoader(arrayOf(File("build/lib/My-process-1.0.jar").toURI().toURL()))
> > val clazz = classloader.loadClass("com.MyProcess.MyClass")
> > val method = clazz.getMethod("myProcess", String::class.java)
> > val instance =
> > Guice.createInjector(GroovyModule()).getInstance(clazz)
> > method.invoke(instance, "")
> > * If I run this in a loop 1000's of times and I profile it, I see that
> > there are a lot of
> > org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry entries in the
> > heap
> > * This steadily increases my memory foot print
> > * I also noticed that if I dont create the class instance, but instead only
> > load and unload the jar through a class loader, I dont see a memory leak
> > * This leads me to think that this piece of code auto injected in the
> > constructor has something to do with it?
> > public MyClass() {
> > MetaClass var2 = this.$getStaticMetaClass();
> > this.metaClass = var2;
> > }
> >
> > Any help is appreciated. I dont use dynamic invocation so I have no use for
> > this metaclass. I compile all my scripts statically
> >
> > regards
> > Saravanan
> >
>
Re: Dynamic class types
Worked like a charm. I used my derived groovyclassloader to create my own compile unit that overrides the default resolve visitor with my own resolve visitor that first resolves per my needs and then calls the default resolve visitor to close out every else. Thank you for your response Regards On Mon, 19 Oct, 2020, 21:32 Milles, Eric (TR Technology), < eric.mil...@thomsonreuters.com> wrote: > You might run your transformation in (at least) 2 phases. In the first > phase, you can mark or replace ClassNode instances that you expect to fill > in as resolved or primary. ResolveVisitor checks isResolved() and > isPrimaryClassNode() first. > > -Original Message- > From: Saravanan Palanichamy > Sent: Monday, October 19, 2020 5:53 AM > To: dev@groovy.apache.org > Subject: Dynamic class types > > Hello everyone > > My DSL that uses AST transformations has this quirk > * A Groovy script defines classes and uses other classes from the class > path > * I want to be able to replace all these types during compile with my own > standard class definition > * If I have the types I am replacing in the class path, everything works. > If I do not have them in the class path, then the resolvevisitor fails > trying to locate these classes. This is as expected. In my case though, I > have no use for these classes after the compile, I replace all of these > classes with a standard class definition and route all data and method > access through this standard class definition > > My question is this > * How can I create a place holder definition for these external classes so > things will load fine, giving me a chance to replace them with the standard > definition and cleanly tie up loose ends through an AST transformation? I > am looking for a mechanism similar to the type checking extensions, > essentially a type specifying extension that works in the semantic analysis > stage > > What I have tried so far > * I use a custom parent class loader that will return this standard > definition for any classes not found in the parent. But this did not work > because the class loading attempts to search through all standard package > paths and I end up seeing inner lookups, nested lookups and outer lookups > all of which should fail. > > Any help on how to proceed is super appreciated >
Dynamic class types
Hello everyone My DSL that uses AST transformations has this quirk * A Groovy script defines classes and uses other classes from the class path * I want to be able to replace all these types during compile with my own standard class definition * If I have the types I am replacing in the class path, everything works. If I do not have them in the class path, then the resolvevisitor fails trying to locate these classes. This is as expected. In my case though, I have no use for these classes after the compile, I replace all of these classes with a standard class definition and route all data and method access through this standard class definition My question is this * How can I create a place holder definition for these external classes so things will load fine, giving me a chance to replace them with the standard definition and cleanly tie up loose ends through an AST transformation? I am looking for a mechanism similar to the type checking extensions, essentially a type specifying extension that works in the semantic analysis stage What I have tried so far * I use a custom parent class loader that will return this standard definition for any classes not found in the parent. But this did not work because the class loading attempts to search through all standard package paths and I end up seeing inner lookups, nested lookups and outer lookups all of which should fail. Any help on how to proceed is super appreciated
Re: RE: CompareToNullExpression does not visitTransforms
On 2020/09/08 14:59:46, "Milles, Eric (TR Technology)" wrote: > https://github.com/apache/groovy/pull/1361 > Thank you Eric, I'll give this a shot
CompareToNullExpression does not visitTransforms
Hello
I am running into an issue with the ClassCodeExpressionTransformer. Why does
CompareToNullExpression not visit any expressions it owns? Should it not
transform objectExpression? I am trying to change property access into my
custom method calls and if a compare to null expression with property access is
used as part of an if condition, it does not transform it
if (a.b == null) {
}
I am trying to change this to
if (a.get("b") == null) {}
The AST works out to a booleanexpression holding a CompareToNullExpression with
a property expression. When I visit this AST with a transformer, it does not
try to transform the property expression
Any help is appreciated
regards
Saravanan
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Hello Daniel
I was able to reproduce this error. I am still not sure if it is my setup
that is causing this error. I have attached my code files as a zip file
My class loader code is this. Notice that I overload the loadClass method
and if I hit a class not found exception, I try to parse the class through
a second parseClass method.
The problem I see is this. I am importing another class (Pojo) from the
first groovy file. So when I compile the first file, internally it tries to
resolve Pojo and tries to compile the second file. Which is where it hits
the getAndPut call in sourceCache.
This works in 2.5 because it is using a concurrentcommoncache for
sourceCache, whereas 3.0.5 is using a StampedCommonCache which does not
allow recursion on getAndPut. The logs I see in my console show I am
attempting to load the top most class and its import class one after the
other. My unit test is also added towards the end of this file
public class MyClassLoader extends GroovyClassLoader {
private FlowRepository repository;
private CompilerConfiguration configuration;
public MyClassLoader(ClassLoader parentClassLoader,
CompilerConfiguration configuration, FlowRepository repository) {
super(parentClassLoader, configuration);
this.repository = repository;
}
@Override
public Class loadClass(
String name,
boolean lookupScriptFiles,
boolean preferClassOverScript,
boolean resolve
) throws ClassNotFoundException, CompilationFailedException {
try {
// Never try to load from script files, always parse from repository
return super.loadClass(name, false, true, resolve);
} catch (ClassNotFoundException e) {
// Try fetching it from the repository
return parseFromRepository(name);
}
}
/**
* Parse [name] from the flow repository and cache it
*/
private Class parseFromRepository(String name) throws
UnsupportedOperationException, ClassNotFoundException {
// Fetch flow source
Flow flow = repository.fetchFlow(name);
if (flow == null)
throw new ClassNotFoundException("Unable to find flow $name");
// Parse the class if found
try {
System.out.println("Loading class " + flow.sourceCode.getName());
// If parseClass yields a class whose name does not match
(typically when we are trying to find an inner
// class (A.B) by parsing the outer class source file
(A.groovy), throw class not found. The compiler will
// attempt to then find A$B which is how inner classes are
stored in the class loader. This find will
// succeed (this process is repeated for inner classes
inside inner classes)
super.parseClass(flow.sourceCode, true);
System.out.println("Finished loading class " +
flow.sourceCode.getName());
// The class parsed may not be the class you are looking
for, but it may be an inner class, so
// use the cache to get it
Class cachedClass = super.getClassCacheEntry(name);
if (cachedClass != null)
return cachedClass;
else
throw new ClassNotFoundException("Unable to find flow $name");
} catch (CompilationFailedException e) {
throw new UnsupportedOperationException("Compile error", e);
}
}
}
@Test
fun testMe() {
val repository = FlowFolderRepository()
val config = CompilerConfiguration().addCompilationCustomizers(
ASTTransformationCustomizer(CompileStatic::class.java)
)
val sandbox = MyClassLoader(this::class.java.classLoader, config,
repository)
// Load all flows in the repository
for (flow in repository.fetchAllFlows()) {
// Make sure class is not already loaded in the parent class
loader. This is usually a Config file error
try {
this::class.java.classLoader.loadClass(flow.name)
throw ConfigQualityException("${flow.name} is already
loaded and cannot be recompiled.")
} catch (e: ClassNotFoundException) {
}
// Load the class
sandbox.loadClass(flow.name)
}
}
On Mon, Aug 10, 2020 at 10:11 AM Daniel Sun wrote:
> Could you provide a standalone runnable script to reproduce the issue?
> BTW, the script is expected to be written in Groovy/Java.
>
> Cheers,
> Daniel Sun
> On 2020/08/08 13:19:01, Saravanan Palanichamy wrote:
> > Hello
> >
> > I have an extension of GroovyClassLoader that I use as follows
> >
> > a) I override loadClass to make sure I decide where to load from
> > b) I first call loadClass on my parent class loader to make sure the
> class is not loaded there (I assert if it is. This code is not shown below)
> > c) I then call loadClass after turning
Re: Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Apologies to the group
This was a bad dependency merge of junit that was causing this thread lockup.
It works just fine now
regards
Saravanan
On 2020/08/10 04:41:42, Daniel Sun wrote:
> Could you provide a standalone runnable script to reproduce the issue?
> BTW, the script is expected to be written in Groovy/Java.
>
> Cheers,
> Daniel Sun
> On 2020/08/08 13:19:01, Saravanan Palanichamy wrote:
> > Hello
> >
> > I have an extension of GroovyClassLoader that I use as follows
> >
> > a) I override loadClass to make sure I decide where to load from
> > b) I first call loadClass on my parent class loader to make sure the class
> > is not loaded there (I assert if it is. This code is not shown below)
> > c) I then call loadClass after turning off all script loading on
> > myclassloader to make sure this class is not already loaded. If it is, I
> > just use that
> > d) If I get a class not found exception, I know this class is not already
> > loaded and so I look up a local repository to fetch a string based script
> > and call parseClass (I do all of this in the context of loadClass)
> >
> > This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a
> > blocked state (not sure if it is blocked on some mutex because of a
> > recursion some where). When I use my debugger , when it gets to this state,
> > I am not able to read any of my variable values in the debugger (which
> > seems to indicate some threading read lock somewhere)
> >
> > Question :- Am I doing this the right way? Why does it block and not load
> > the class? It locks up in the parseClass invocation (inside a validate call
> > that validates the sourceCode object
> >
> > My code looks like this
> >
> > class MyClassLoader constructor(
> > private val repository: FlowsRepository,
> > private val configuration: CompilerConfiguration,
> > parentClassLoader: ClassLoader
> > ) : GroovyClassLoader(parentClassLoader, configuration) {
> >
> > /** Load class [name] either from repository or class path */
> > override fun loadClass(
> > name: String,
> > lookupScriptFiles: Boolean,
> > preferClassOverScript: Boolean,
> > resolve: Boolean
> > ): Class<*> {
> > return try {
> > // Never try to load from script files, always parse from
> > repository
> > super.loadClass(name, false, true, resolve)
> > } catch (e: ClassNotFoundException) {
> > // Try fetching it from the repository
> > parseFromRepository(name)
> > }
> > }
> >
> > /** Parse [name] from the flow repository and cache it */
> > private fun parseFromRepository(name: String): Class<*> {
> >
> > // Fetch flow source
> > val flow = repository.fetchFlow(name) ?: throw
> > ClassNotFoundException("Unable to find flow $name")
> >
> > // Parse the class if found
> > try {
> > // If parseClass yields a class whose name does not match
> > (typically when we are trying to find an inner
> > // class (A.B) by parsing the outer class source file
> > (A.groovy), throw class not found. The compiler will
> > // attempt to then find A$B which is how inner classes are
> > stored in the class loader. This find will
> > // succeed (this process is repeated for inner classes inside
> > inner classes)
> > super.parseClass(flow.sourceCode, true)
> >
> > // The class parsed may not be the class you are looking for,
> > but it may be an inner class, so
> > // use the cache to get it
> > return getClassCacheEntry(name) ?: throw
> > ClassNotFoundException("Unable to find flow $name")
> > } catch (e: CompilationFailedException) {
> > throw CodeQualityException("Compile error", e)
> > }
> > }
> > }
> >
>
Re: Parrot parser customization
Hello Daniel
Thank you for your email. I am of the impression that the AST transformation
works on a valid AST. I am trying to define custom syntax to make a valid
custom AST. This means I am creating new syntax that is not easy to represent
using the groovy language spec. Is that the right way to think about this or
are you saying that it should be possible to use fluent call mechanims and
chaining to get what I want. I am not sure if that is easy to achieve because
the syntax for my custom query can get hairy
regards
Saravanan
On 2020/08/09 01:24:57, Daniel Sun wrote:
> I think you want to implement your own DSL. The AST transformation of Groovy
> could help you.
>
> Cheers,
> Daniel Sun
> On 2020/08/08 23:54:47, Saravanan Palanichamy wrote:
> > Hello everyone
> >
> > If I wanted to introduce new syntax in my groovy script, how would I go
> > about doing it? I want to embed custom syntax directly into the groovy file
> > and have it be parsed into my custom AST nodes. An example would be
> >
> > myFunction() {
> >List tableValues = select value from mySQLTable where
> > (select tableName from myTableNames where user = $userName)
> >
> > ... Use table Values
> > }
> >
> > I want to enable a few behaviors
> > a) Check for syntax correctness (which is why I want to parse using antlr
> > grammar)
> > b) Check for semantic correctness (I suppose if I parsed this into my
> > custom AST nodes, that takes care of that. I could make an
> > SQLExpressionASTNode and validate things there)
> > c) Enable a debug experience where I am able to see the result of the inner
> > SQL first and then see it move to the outer SQL (this would be super
> > awesome, but I realize this is in the purview of the groovy IDE plugin. I
> > am asking here to see if I can get any pointers)
> >
> > My limited ideas so far are to annotate the List declaration with @SQL,
> > hook into the semantic phase to translate the select clause (embedded in a
> > gstring) into a validated, redirect into a custom function call.
> >
> > so if I see
> >
> > @SQL List tableValues = "select "
> >
> > I'll convert it to
> > @SQL List tableValues = sqlRunner.run("select ...")
> >
> > This does not get me debuggability though and it feels contrived
> >
> > regards
> > Saravanan
> >
>
Parrot parser customization
Hello everyone
If I wanted to introduce new syntax in my groovy script, how would I go about
doing it? I want to embed custom syntax directly into the groovy file and have
it be parsed into my custom AST nodes. An example would be
myFunction() {
List tableValues = select value from mySQLTable where (select
tableName from myTableNames where user = $userName)
... Use table Values
}
I want to enable a few behaviors
a) Check for syntax correctness (which is why I want to parse using antlr
grammar)
b) Check for semantic correctness (I suppose if I parsed this into my custom
AST nodes, that takes care of that. I could make an SQLExpressionASTNode and
validate things there)
c) Enable a debug experience where I am able to see the result of the inner SQL
first and then see it move to the outer SQL (this would be super awesome, but I
realize this is in the purview of the groovy IDE plugin. I am asking here to
see if I can get any pointers)
My limited ideas so far are to annotate the List declaration with @SQL, hook
into the semantic phase to translate the select clause (embedded in a gstring)
into a validated, redirect into a custom function call.
so if I see
@SQL List tableValues = "select "
I'll convert it to
@SQL List tableValues = sqlRunner.run("select ...")
This does not get me debuggability though and it feels contrived
regards
Saravanan
Upgrading from 2.5 to 3.0.5 and my class loader extension does not work
Hello
I have an extension of GroovyClassLoader that I use as follows
a) I override loadClass to make sure I decide where to load from
b) I first call loadClass on my parent class loader to make sure the class is
not loaded there (I assert if it is. This code is not shown below)
c) I then call loadClass after turning off all script loading on myclassloader
to make sure this class is not already loaded. If it is, I just use that
d) If I get a class not found exception, I know this class is not already
loaded and so I look up a local repository to fetch a string based script and
call parseClass (I do all of this in the context of loadClass)
This used to work with 2.5.x but when I upgrade to 3.0.5, it sits in a blocked
state (not sure if it is blocked on some mutex because of a recursion some
where). When I use my debugger , when it gets to this state, I am not able to
read any of my variable values in the debugger (which seems to indicate some
threading read lock somewhere)
Question :- Am I doing this the right way? Why does it block and not load the
class? It locks up in the parseClass invocation (inside a validate call that
validates the sourceCode object
My code looks like this
class MyClassLoader constructor(
private val repository: FlowsRepository,
private val configuration: CompilerConfiguration,
parentClassLoader: ClassLoader
) : GroovyClassLoader(parentClassLoader, configuration) {
/** Load class [name] either from repository or class path */
override fun loadClass(
name: String,
lookupScriptFiles: Boolean,
preferClassOverScript: Boolean,
resolve: Boolean
): Class<*> {
return try {
// Never try to load from script files, always parse from repository
super.loadClass(name, false, true, resolve)
} catch (e: ClassNotFoundException) {
// Try fetching it from the repository
parseFromRepository(name)
}
}
/** Parse [name] from the flow repository and cache it */
private fun parseFromRepository(name: String): Class<*> {
// Fetch flow source
val flow = repository.fetchFlow(name) ?: throw
ClassNotFoundException("Unable to find flow $name")
// Parse the class if found
try {
// If parseClass yields a class whose name does not match
(typically when we are trying to find an inner
// class (A.B) by parsing the outer class source file (A.groovy),
throw class not found. The compiler will
// attempt to then find A$B which is how inner classes are stored
in the class loader. This find will
// succeed (this process is repeated for inner classes inside inner
classes)
super.parseClass(flow.sourceCode, true)
// The class parsed may not be the class you are looking for, but
it may be an inner class, so
// use the cache to get it
return getClassCacheEntry(name) ?: throw
ClassNotFoundException("Unable to find flow $name")
} catch (e: CompilationFailedException) {
throw CodeQualityException("Compile error", e)
}
}
}
Re: StaticPropertyAccessHelper transforms are not routed through my expression transformer
Thank you Eric. Let me know how I can help.
regards
Saravanan
On Thu, Aug 6, 2020 at 7:35 PM Milles, Eric (TR Technology) <
eric.mil...@thomsonreuters.com> wrote:
> Your assessment appears correct.
>
> PropertyExpression works like this:
> public Expression transformExpression(ExpressionTransformer
> transformer) {
> PropertyExpression ret = new
> PropertyExpression(transformer.transform(objectExpression),
> transformer.transform(property), safe);
>
> PoppingMethodCallExpression works like this:
> public Expression transformExpression(final ExpressionTransformer
> transformer) {
> PoppingMethodCallExpression call = new
> PoppingMethodCallExpression(getObjectExpression().transformExpression(transformer),
> getMethodTarget(), (TemporaryVariableExpression)
> tmp.transformExpression(transformer));
>
>
> There may be several instances of this to correct and test if all of the
> Expression sub-types are looked at. I'd start by filing a new Jira ticket.
>
>
> -Original Message-
> From: Saravanan Palanichamy
> Sent: Thursday, August 6, 2020 8:43 AM
> To: dev@groovy.apache.org
> Subject: StaticPropertyAccessHelper transforms are not routed through my
> expression transformer
>
> Hello
>
> My class derives from ClassCodeExpressionTransformer and I use it to
> transform my method call expression. Lets say I add an extra parameter to
> these methods. This works for normal expression, however for property
> setter expressions, this does not work because it does not route through my
> class
>
> I root caused this to the StaticPropertyAccessHelper class which uses a
> PoppingMethodCallExpression. In this class, it does not call
> transformer.transform(receiver), instead it invokes
> receiver.transformExpression(transformer). This means code like this
>
> myObj.property = myFunction()
>
> Where property is a setProperty() function will result in not routing
> through my transformer. Am I root causing this correctly or is my
> understanding wrong? What is the fix for this? I am using Groovy 2.5.x. Is
> this fixed in newer versions?
>
> regards
> Saravanan
>
StaticPropertyAccessHelper transforms are not routed through my expression transformer
Hello My class derives from ClassCodeExpressionTransformer and I use it to transform my method call expression. Lets say I add an extra parameter to these methods. This works for normal expression, however for property setter expressions, this does not work because it does not route through my class I root caused this to the StaticPropertyAccessHelper class which uses a PoppingMethodCallExpression. In this class, it does not call transformer.transform(receiver), instead it invokes receiver.transformExpression(transformer). This means code like this myObj.property = myFunction() Where property is a setProperty() function will result in not routing through my transformer. Am I root causing this correctly or is my understanding wrong? What is the fix for this? I am using Groovy 2.5.x. Is this fixed in newer versions? regards Saravanan
Re: Execute a code segment during script compile
Just to close out this discussion. The way I proceeded was to finish the
compile, then invoke the compiled closure (if it did not need any parameters)
to get the created object. I can then analyze the object to pull out the
constants needed.
Thanks for all your responses.
On 2020/02/25 09:19:58, Saravanan Palanichamy wrote:
> :D yes which is why I am using the script to gather intent and not using it
> as the runtime. I need to understand the retry parameters of 10 and 20. I
> gather this knowledge at the time of creation and compilation which is
> harmless to exits
>
> On 2020/02/24 08:15:38, Alessio Stalla wrote:
> > You wouldn't want to run @Retry({System.exit(0)}) on your server, I presume.
> >
> > On Mon, 24 Feb 2020 at 00:39, Saravanan Palanichamy
> > wrote:
> >
> > >
> > >
> > > On 2020/02/23 23:14:32, Paul King wrote:
> > > > Just for future reference, I'd probably start out with such a question
> > > > on
> > > > the users mailing list. There are more folks subscribed to that list and
> > > > writing closures and transforms (using Groovy) are topics which that
> > > > list
> > > > covers. If it turned out that Groovy couldn't handle your use case, the
> > > dev
> > > > list (developing the language) would be the place to go to ask whether a
> > > > feature could be added to the language.
> > > >
> > > > Having said that, to answer your question, there are quite a lot of
> > > > things that are possible. Perhaps you could give a concrete simple
> > > example
> > > > of the kind of thing you are trying to do. I understand most of what you
> > > > are saying but a few bits are still a little unclear (to me at least).
> > > >
> > > > Cheers, Paul.
> > > >
> > > >
> > > > On Mon, Feb 24, 2020 at 9:06 AM Saravanan Palanichamy <
> > > chava...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > Is it possible to do this in the groovy AST transformation ->
> > > > >
> > > > > a) in a code visitor, visit a closure expression (in the
> > > > > INSTRUCTION_SELECTION phase)
> > > > > b) Using the Closure Node, execute this code to determine its results
> > > > > based on different parameters
> > > > >
> > > > > Essentially I want to be able to selectively run a closure defined in
> > > code
> > > > > during the compile process. I see you can convert closures into
> > > strings, is
> > > > > it possible to compile that string in the middle of a compile process?
> > > > >
> > > > > I am not sure this is the right forum for this question, please let me
> > > > > know if otherwise
> > > > >
> > > >
> > >
> > > Hello Paul
> > >
> > > Thank you for your reply. I am trying to do this specific thing
> > > a) I allow my developers to write Groovy scripts
> > > b) I use the script as a DSL to generate actual configuration files needed
> > > for my service at runtime.
> > > c) Because I need to translate the code to configuration entries, I need
> > > to execute some parts of the script to determine config values. For
> > > example
> > >
> > > @Retry({new RetryParams(10, 20)})
> > > void runSomeCodeInMyServer() {
> > > }
> > >
> > > In the code above, lets say this code runs on my server, but the server
> > > needs to be told that the parameters for the thread that executes this is
> > > to retry it 10 times, with a 20 second interval.
> > >
> > > When parsing the groovy script, I need to pull out these 10 and 20 numbers
> > > from the script. One way to do this is visit the closure, see this was a
> > > constructor call to RetryParams, and use the numbers there. This seems
> > > tedious given the number of combinations possible. However if there was
> > > way
> > > for me to run this closure during the AST transform, I can inspect the
> > > created retryparams object to get my values. Does that make sense?
> > >
> > > As I am typing this, I also realize that the closure may call other
> > > functions which may need to be compiled as well (but I can create compile
> > > errors there to keep it simple)
> > >
> >
>
Re: Execute a code segment during script compile
:D yes which is why I am using the script to gather intent and not using it as
the runtime. I need to understand the retry parameters of 10 and 20. I gather
this knowledge at the time of creation and compilation which is harmless to
exits
On 2020/02/24 08:15:38, Alessio Stalla wrote:
> You wouldn't want to run @Retry({System.exit(0)}) on your server, I presume.
>
> On Mon, 24 Feb 2020 at 00:39, Saravanan Palanichamy
> wrote:
>
> >
> >
> > On 2020/02/23 23:14:32, Paul King wrote:
> > > Just for future reference, I'd probably start out with such a question on
> > > the users mailing list. There are more folks subscribed to that list and
> > > writing closures and transforms (using Groovy) are topics which that list
> > > covers. If it turned out that Groovy couldn't handle your use case, the
> > dev
> > > list (developing the language) would be the place to go to ask whether a
> > > feature could be added to the language.
> > >
> > > Having said that, to answer your question, there are quite a lot of
> > > things that are possible. Perhaps you could give a concrete simple
> > example
> > > of the kind of thing you are trying to do. I understand most of what you
> > > are saying but a few bits are still a little unclear (to me at least).
> > >
> > > Cheers, Paul.
> > >
> > >
> > > On Mon, Feb 24, 2020 at 9:06 AM Saravanan Palanichamy <
> > chava...@gmail.com>
> > > wrote:
> > >
> > > > Hello
> > > >
> > > > Is it possible to do this in the groovy AST transformation ->
> > > >
> > > > a) in a code visitor, visit a closure expression (in the
> > > > INSTRUCTION_SELECTION phase)
> > > > b) Using the Closure Node, execute this code to determine its results
> > > > based on different parameters
> > > >
> > > > Essentially I want to be able to selectively run a closure defined in
> > code
> > > > during the compile process. I see you can convert closures into
> > strings, is
> > > > it possible to compile that string in the middle of a compile process?
> > > >
> > > > I am not sure this is the right forum for this question, please let me
> > > > know if otherwise
> > > >
> > >
> >
> > Hello Paul
> >
> > Thank you for your reply. I am trying to do this specific thing
> > a) I allow my developers to write Groovy scripts
> > b) I use the script as a DSL to generate actual configuration files needed
> > for my service at runtime.
> > c) Because I need to translate the code to configuration entries, I need
> > to execute some parts of the script to determine config values. For example
> >
> > @Retry({new RetryParams(10, 20)})
> > void runSomeCodeInMyServer() {
> > }
> >
> > In the code above, lets say this code runs on my server, but the server
> > needs to be told that the parameters for the thread that executes this is
> > to retry it 10 times, with a 20 second interval.
> >
> > When parsing the groovy script, I need to pull out these 10 and 20 numbers
> > from the script. One way to do this is visit the closure, see this was a
> > constructor call to RetryParams, and use the numbers there. This seems
> > tedious given the number of combinations possible. However if there was way
> > for me to run this closure during the AST transform, I can inspect the
> > created retryparams object to get my values. Does that make sense?
> >
> > As I am typing this, I also realize that the closure may call other
> > functions which may need to be compiled as well (but I can create compile
> > errors there to keep it simple)
> >
>
Re: Execute a code segment during script compile
On 2020/02/23 23:14:32, Paul King wrote:
> Just for future reference, I'd probably start out with such a question on
> the users mailing list. There are more folks subscribed to that list and
> writing closures and transforms (using Groovy) are topics which that list
> covers. If it turned out that Groovy couldn't handle your use case, the dev
> list (developing the language) would be the place to go to ask whether a
> feature could be added to the language.
>
> Having said that, to answer your question, there are quite a lot of
> things that are possible. Perhaps you could give a concrete simple example
> of the kind of thing you are trying to do. I understand most of what you
> are saying but a few bits are still a little unclear (to me at least).
>
> Cheers, Paul.
>
>
> On Mon, Feb 24, 2020 at 9:06 AM Saravanan Palanichamy
> wrote:
>
> > Hello
> >
> > Is it possible to do this in the groovy AST transformation ->
> >
> > a) in a code visitor, visit a closure expression (in the
> > INSTRUCTION_SELECTION phase)
> > b) Using the Closure Node, execute this code to determine its results
> > based on different parameters
> >
> > Essentially I want to be able to selectively run a closure defined in code
> > during the compile process. I see you can convert closures into strings, is
> > it possible to compile that string in the middle of a compile process?
> >
> > I am not sure this is the right forum for this question, please let me
> > know if otherwise
> >
>
Hello Paul
Thank you for your reply. I am trying to do this specific thing
a) I allow my developers to write Groovy scripts
b) I use the script as a DSL to generate actual configuration files needed for
my service at runtime.
c) Because I need to translate the code to configuration entries, I need to
execute some parts of the script to determine config values. For example
@Retry({new RetryParams(10, 20)})
void runSomeCodeInMyServer() {
}
In the code above, lets say this code runs on my server, but the server needs
to be told that the parameters for the thread that executes this is to retry it
10 times, with a 20 second interval.
When parsing the groovy script, I need to pull out these 10 and 20 numbers from
the script. One way to do this is visit the closure, see this was a constructor
call to RetryParams, and use the numbers there. This seems tedious given the
number of combinations possible. However if there was way for me to run this
closure during the AST transform, I can inspect the created retryparams object
to get my values. Does that make sense?
As I am typing this, I also realize that the closure may call other functions
which may need to be compiled as well (but I can create compile errors there to
keep it simple)
Execute a code segment during script compile
Hello Is it possible to do this in the groovy AST transformation -> a) in a code visitor, visit a closure expression (in the INSTRUCTION_SELECTION phase) b) Using the Closure Node, execute this code to determine its results based on different parameters Essentially I want to be able to selectively run a closure defined in code during the compile process. I see you can convert closures into strings, is it possible to compile that string in the middle of a compile process? I am not sure this is the right forum for this question, please let me know if otherwise
