jerryshao commented on PR #9588:
URL: https://github.com/apache/gravitino/pull/9588#issuecomment-3780716245
> The main issue is determining who is responsible for escaping special
characters when submitting a job: the job client or the job executor? I prefer
the latter option, as escaping special characters is part of the internal logic
for local shell executors. What do you think? Below is an example of a job
configuration for submitting a job.
>
> Job config items Job config values Example
> where_clause normal string a = 1 and b = "2"
> options json str {"target-file-size-bytes":"536870912"}
> spark_conf json str {"spark.sql.shuffle.partitions":"200"}
I think this is not a client or executor problem, it is the problem of how
Java handles `"` in the input argv, no matter whether the program is run by CLI
or programmatically.
According to my test in IDEA, the input parameter `a = 1 and b = "2"` will
be converted to `a = 1 and b = 2` when getting from argv, the " is lost, which
means the meaing of string literal is changed. Also for
`{"target-file-size-bytes":"536870912"}`, it will be converted to
`{target-file-size-bytes:536870912}`, which is not a json map anymore.
If we input `a = 1 and b = \"2\"`, then it will convert to `a = 1 and b =
"2"` in the argv. And for `{\"target-file-size-bytes\":\"536870912\"}`, it will
convert to `{"target-file-size-bytes":"536870912"}`, which is a json map. This
is what we expected.
What I want to say is that the handling of escaping `"` is not controlled
and cannot be controlled by this code I implemented. This code doesn't know
whether the where clause is a string literal or an integer.
Besides, the input parameters may be different if you run in different
environments (CLI, IDEA, Java program).
What this code **wants** is a correct string, whether it is a JSON string or
a string literal. It is the user's responsibility to make sure the input
parameters can be correctly converted to the expected string, not the code here.
I suggest you try this with your TMS framework, you will better know how to
use this job. If there's a problem, then we can fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
