Murat ODUNC created BEAM-5966:
---------------------------------
Summary: When working "LiteralGqlQuery", Error: Query cannot have
any sort orders.
Key: BEAM-5966
URL: https://issues.apache.org/jira/browse/BEAM-5966
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.8.0
Reporter: Murat ODUNC
Assignee: Chamikara Jayalath
Hello..
I have used Apache Beam pipelines on my project. And I have a problem to read
data from GCP DataStore.
My First question is "How to create "Query" object" . I'am sharing a sample of
regarding JavaDoc.
[https://beam.apache.org/releases/javadoc/2.8.0/]
{code:java}
Query query = ...; // I dont know any idea how configure the object
String projectId = "...";
Pipeline p = Pipeline.create(options);
PCollection<Entity> entities = p.apply(
DatastoreIO.v1().read()
.withProjectId(projectId)
.withQuery(query));
{code}
My second question is how to set "sort oder" of the query?
I tried to 'LiteralGqlQuery' to read data but DataStore IO Reader failed by
follow error trace..
{noformat}
java.lang.IllegalArgumentException: Query cannot have any sort orders. at
com.google.datastore.v1.client.QuerySplitterImpl.validateQuery(QuerySplitterImpl.java:128)
~[datastore-v1-proto-client-1.6.0.jar:na] at
com.google.datastore.v1.client.QuerySplitterImpl.getSplits(QuerySplitterImpl.java:69)
~[datastore-v1-proto-client-1.6.0.jar:na] at
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read.splitQuery(DatastoreV1.java:454)
~[beam-sdks-java-io-google-cloud-platform-2.8.0.jar:na] at
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read.access$100(DatastoreV1.java:264)
~[beam-sdks-java-io-google-cloud-platform-2.8.0.jar:na] at
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read$SplitQueryFn.processElement(DatastoreV1.java:813)
~[beam-sdks-java-io-google-cloud-platform-2.8.0.jar:na] at
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read$SplitQueryFn$DoFnInvoker.invokeProcessElement(Unknown
Source) [na:na] at
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
[beam-runners-direct-java-2.8.0.jar:na] at
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
[beam-runners-direct-java-2.8.0.jar:na] at
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:78)
[beam-runners-direct-java-2.8.0.jar:na]
at
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:207)
[beam-runners-direct-java-2.8.0.jar:na] at
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:55)
[beam-runners-direct-java-2.8.0.jar:na] at
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
[beam-runners-direct-java-2.8.0.jar:na] at
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
[beam-runners-direct-java-2.8.0.jar:na] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_162] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_162] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_162] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
{noformat}
My code is here
{code:java}
PipelineOptions options = PipelineOptionsFactory.create();
options.setRunner(DirectRunner.class); String gqlQuery = String.format("SELECT
* FROM task " + "WHERE createdAt > DATETIME('%s') "+ "AND createdAt <
DATETIME('%s') ORDER BY createdAt ASC", "2018-11-03T00:00:00Z",
"2018-11-03T23:59:59Z" ); log.info("GQL: " + gqlQuery); String projectId =
"bonair-pass"; String ns = "prod-01ce205a-22ff-4bab-b133-926bdc54c8b3";
Pipeline p = Pipeline.create(options);
PCollection<String> entities = p.apply( DatastoreIO.v1().read()
.withProjectId(projectId).withLiteralGqlQuery(gqlQuery) .withQuery(query)
.withNamespace(ns)) .apply("Debug", ParDo.of(new DoFn<Entity, String>(){
@ProcessElement public void processElement(ProcessContext c) {
//log.info(c.element().getProperties().get("foo").getStringValue()); } }) );
p.run().waitUntilFinish();
{code}
```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
